CN115859278B - Method, system, equipment and storage medium for auditing software operation behaviors - Google Patents
Method, system, equipment and storage medium for auditing software operation behaviors Download PDFInfo
- Publication number
- CN115859278B CN115859278B CN202310181296.0A CN202310181296A CN115859278B CN 115859278 B CN115859278 B CN 115859278B CN 202310181296 A CN202310181296 A CN 202310181296A CN 115859278 B CN115859278 B CN 115859278B
- Authority
- CN
- China
- Prior art keywords
- window
- operation interface
- application software
- interface
- software
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a method, a system, equipment and a storage medium for auditing software operation behaviors, wherein the method comprises the following steps: capturing a plurality of operation interfaces of application software of a target terminal; sequentially carrying out window segmentation on the operation interface to obtain window information of each window in the operation interface; determining the attribute of the application software of the operation interfaces according to the window information, and determining the change track of the operation interfaces based on the window information of each operation interface; tracking operation behaviors executed by a computing user according to the attribute and the operation interface change track; and performing audit analysis on the operation behavior. According to the method, window segmentation is carried out on the multiple grabbed operation interfaces, so that window information of each window in the operation interfaces is obtained; and determining the attribute of the application software according to the window information, and tracking the operation behavior executed by the user according to the attribute and the change track of the operation interface.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a method, a system, an apparatus, and a storage medium for auditing software operation behaviors.
Background
With the rapid development of internet information technology, various information systems and network products are layered endlessly. The wide application of various information systems and network products is a double-edged sword, which brings about standard, convenient and efficient office flow and business modes on one hand, and causes the problems of safety, difficult control of internal operation and maintenance, difficult tracing and the like on the other hand. These problems threaten the security of the information center. Such as: internal business data are tampered, revealed and stolen; access to illegal websites at servers, misoperations, messy operations on important servers, etc. How to effectively operate and maintain audit on an information center in an enterprise has become an important problem for governments, finances and enterprises to face.
At present, two main methods are adopted for the server operation and maintenance audit scheme: firstly, the audit software is installed in the computer operating system, and then the operating behavior of the computer software is obtained by using an API provided by the operating system. For example, the window currently operated by the user can be obtained by using getforegroudwindow on Windows, then other information of the window is further obtained through an API, for example, a process ID is obtained through GetWindowThreadProcessId (then a process name is obtained through the process ID), for example, title text of the window is obtained through GetWindowText, and all controls in the window are enumerated through engmchildwindows. However, when the application software is not a used Windows control, the operation behavior of the software is difficult to obtain in an API mode, and different operating systems need to be re-adapted, so that the scheme has great compatibility problem and great development cost. Secondly, the mode of grabbing the screen to display images or record the screen is used, so that the challenge brought by the mode is that when the number of audited computers is too large, the occupied storage space is large, the audited computers can only see massive images or record the screen, the audit labor investment is large, and the applicability is reduced.
Therefore, a method for auditing the operation behaviors of software is needed, which not only can audit the operation behaviors of application software of various operation systems, but also can improve the auditing efficiency.
Disclosure of Invention
The invention mainly aims to provide a software operation behavior auditing method, a system, equipment and a storage medium, and aims to solve the technical problems that in the prior art, auditing of application software operation behaviors is limited by the type of an operation system and auditing efficiency is low.
In order to achieve the above object, the present invention provides a software operation behavior auditing method, the method comprising the steps of:
capturing a plurality of operation interfaces of application software of a target terminal;
sequentially carrying out window segmentation on the operation interface to obtain window information of each window in the operation interface;
determining the attribute of the application software of the operation interface according to the window information, and determining the change track of the operation interface based on the window information of each operation interface;
tracking operation behaviors executed by a computing user according to the attributes and the operation interface change track;
and carrying out audit analysis on the operation behaviors.
Optionally, the window information includes a window title, a window text, a window control, a window icon and a window layout, and the step of determining the attribute of the application software of the operation interface according to the window information specifically includes:
and determining the attribute of the application software of the operation interface according to the window title, the window text, the window control, the window icon and the window layout based on a preset application software matching algorithm.
Optionally, the step of grabbing multiple operation interfaces of the application software of the target terminal includes:
sequentially capturing a plurality of operation interfaces of application software of the target terminal by installing audit software on the target terminal;
or alternatively, the first and second heat exchangers may be,
shooting or continuously shooting a display screen of a target terminal at preset intervals by a camera to obtain a plurality of operation interfaces of application software of the target terminal;
or alternatively, the first and second heat exchangers may be,
a conversion device for converting an analog signal into a digital signal is installed on a display signal line of a target terminal, a video display signal is obtained, and the video display signal is converted into a plurality of operation interfaces of application software of the target terminal.
Optionally, the operation interface change track includes a change of window characters and window layouts in an operation interface, and the step of tracking operation behaviors executed by a computing user according to the application software attribute and the operation interface change track includes:
presuming the operation type currently executed by the user according to the attribute of the application software;
according to the change of window characters and window layout in the operation interface, tracking and calculating an operation sequence and operation contents of a user based on the operation type and a preset tracking algorithm;
and recording operation behaviors executed by the user according to the operation sequence and the operation content.
Optionally, after the step of tracking the operation behavior executed by the computing user according to the attribute and the operation interface change track, the method further includes:
acquiring the system time and the corresponding operation type of the user operation sequence according to the operation behavior;
according to the operation type and the operation sequence, an operation interface before application software switching in the operation interface and an operation interface after application software switching in the operation interface are obtained and used as window switching screenshot;
and storing the system time, the operation type, the operation content and the window switching screenshot into a database.
Optionally, the step of sequentially performing window segmentation on the operation interface to obtain window information of each window in the operation interface includes:
identifying each window area according to the RGB value of each operation interface, and obtaining the window layout of the operation interface;
sequentially carrying out window segmentation on each operation interface based on the window layout to obtain a plurality of windows;
and identifying window titles, window characters, window icons and window controls in each window through an optical character identification technology and an image matching algorithm.
Optionally, the attribute of the application software includes a type, a purpose, a name and a version of the application software, and the step of performing audit analysis on the operation behavior specifically includes:
acquiring the switching condition of application software within the preset time range of the target terminal;
counting the use duration of each application software in the preset time range according to the switching condition;
acquiring and counting the specific business behaviors of each application software in the preset time range;
and carrying out audit analysis according to the switching condition, the using time and the specific business behavior.
In addition, to achieve the above object, the present invention also proposes a software operation behavior auditing system, the system comprising:
the interface grabbing module is used for grabbing a plurality of operation interfaces of the application software of the target terminal;
the window segmentation module is used for sequentially carrying out window segmentation on the operation interface to obtain window information of each window in the operation interface;
the information analysis module is used for determining the attribute of the application software of the operation interface according to the window information and determining the change track of the operation interface based on the window information of each operation interface;
the operation restoration module is used for tracking the operation behavior executed by the user according to the attribute and the operation interface change track;
and the audit analysis module is used for carrying out audit analysis on the operation behaviors.
In addition, to achieve the above object, the present invention also proposes a software operation behavior auditing apparatus, the apparatus comprising: the system comprises a memory, a processor and a software operation behavior auditing program stored on the memory and capable of running on the processor, wherein the software operation behavior auditing program is configured to realize the steps of the software operation behavior auditing method.
In addition, in order to achieve the above object, the present invention also proposes a storage medium having stored thereon a software operation behavior auditing program that, when executed by a processor, implements the steps of the software operation behavior auditing method as described above.
The method comprises the steps of capturing a plurality of operation interfaces of application software of a target terminal; then sequentially carrying out window segmentation on the operation interface to obtain window information of each window in the operation interface; determining the attribute of the application software of the operation interface according to the window information, and determining the change track of the operation interface based on the window information of each operation interface; then, according to the attribute and the change track of the operation interface, tracking the operation behavior executed by the user; and finally, performing audit analysis on the operation behavior. According to the method, window segmentation is carried out on the multiple grabbed operation interfaces, so that window information of each window in the operation interfaces is obtained; and determining the attribute of the application software of the operation interface according to the window information, and tracking the operation behavior executed by the user according to the attribute and the change track of the operation interface.
Drawings
FIG. 1 is a schematic diagram of a software operational behavior auditing apparatus of a hardware runtime environment according to an embodiment of the present invention;
FIG. 2 is a flow chart of a first embodiment of the method for auditing the operation behavior of the software of the present invention;
FIG. 3 is a flow chart of a second embodiment of the method for auditing the operation behavior of the software of the present invention;
FIG. 4 is a flowchart of a third embodiment of a method for auditing the behavior of software operations according to the present invention;
fig. 5 is a block diagram of a first embodiment of the software operational behaviour auditing system of the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a software operation behavior audit device of a hardware operation environment according to an embodiment of the present invention.
As shown in fig. 1, the software operation behavior auditing apparatus may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a Wireless interface (e.g., a Wireless-Fidelity (WI-FI) interface). The Memory 1005 may be a high-speed random access Memory (Random Access Memory, RAM) or a stable nonvolatile Memory (NVM), such as a disk Memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
Those skilled in the art will appreciate that the architecture shown in FIG. 1 is not limiting of the software operational behavior auditing apparatus, and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and a software operation behavior auditing program may be included in a memory 1005 as one type of storage medium.
In the software operation behavior auditing apparatus shown in fig. 1, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the software operation behavior auditing device of the present invention may be disposed in the software operation behavior auditing device, where the software operation behavior auditing device invokes a software operation behavior auditing program stored in the memory 1005 through the processor 1001, and executes the software operation behavior auditing method provided by the embodiment of the present invention.
The embodiment of the invention provides a software operation behavior auditing method, and referring to fig. 2, fig. 2 is a flow diagram of a first embodiment of the software operation behavior auditing method.
In this embodiment, the software operation behavior auditing method includes the following steps:
step S10: capturing a plurality of operation interfaces of application software of a target terminal;
it should be noted that, the execution body of the embodiment may be a computing service device with functions of data processing, network communication and program running, such as a server, a tablet computer, a personal computer, a mobile phone, or an electronic device, a software operation behavior auditing device, or the like, which can implement the above functions. This embodiment and the following embodiments are exemplified below by taking a software operation behavior auditing apparatus as an example.
It is understood that the target terminal refers to a computing service device, which may be a personal computer, a tablet computer, a mobile phone, or other electronic devices with similar functions, which is not limited in this embodiment. Application software refers to software used by a user in an electronic device, and application software refers to a software system specifically tailored for a certain application purpose.
In a specific implementation, audit software can be installed on a computer operating system, and then an operation interface of a screen is captured periodically or continuously (such as installation of audit software on windows operating system, automatic running of the audit software after starting up, capturing screenshot of the operation interface every 0.5 seconds); the screen operation interface can be shot regularly or continuously through the camera to obtain operation interface information (for example, the camera is used for shooting the screen of the mobile phone in real time, then key frames are extracted from video streaming, and the key frames are restored to screenshot of the operation interface); the conversion device for converting the analog signal into the digital signal can be arranged on the display signal line to obtain a video display signal, extract an image display picture and finally convert the image display picture into an operation interface screenshot of a screen.
It can be understood that by the above method, the grasping of the application software operation interfaces of different computer systems (whether a desktop operation system or a mobile terminal device) can be realized, and then unified analysis processing is performed to complete the audit of the operation behaviors.
Step S20: sequentially carrying out window segmentation on the operation interface to obtain window information of each window in the operation interface;
it should be noted that, window division refers to dividing a window into multiple panes, where each pane contains views, either the same type of view or different types of views. The window information includes window title, window text, window control, window icon and window layout.
In a specific implementation, firstly, each window area is identified according to RGB values of each operation interface, and window layout of the operation interface is obtained; then, window segmentation is sequentially carried out on each operation interface based on the window layout, and a plurality of windows are obtained; and finally, recognizing window titles, window characters, window icons and window controls in each window through an optical character recognition technology and an image matching algorithm.
Step S30: determining the attribute of the application software of the operation interface according to the window information, and determining the change track of the operation interface based on the window information of each operation interface;
it should be noted that, the attributes of the application software include: the change track of the operation interface comprises the change of window characters in the operation interface, the switching of the application software in the operation interface, the change of icons in the operation interface and the like.
It will be appreciated that the same application software will have the same or similar window and control layout on different operating systems, such as a chrome browser, and that the window layout will be the same whether on window, linux, or macos.
Further, the operation interface is formed by combining a series of windows and controls (such as buttons, progress bars, input frames, title bars, menu bars, tool bars, status bars, document areas … … and the like) according to a certain layout rule, the windows and the controls can contain characters and icons, the purposes of the windows and the controls can be deduced through the contents of the characters and the icons, and therefore the application software to which the operation interface belongs can be obtained through analysis processing of the grabbed operation interface.
It can be appreciated that, after the attribute of the application software to which the operation interface belongs is obtained, it is known to some extent what operation is being performed by the user. For example, the current operation interface is word. Exe program belonging to office, and the user should look at the document or edit text; the current operation interface belongs to a chrome browser, so that the user should be browsing a web page or performing business operation by using an OA system based on a web frame. Further, on the basis of obtaining the attribute of the application software, the change track of the operation interface can be determined based on the window title, the window text, the window control, the window icon and the window layout of each operation interface.
Step S40: tracking operation behaviors executed by a computing user according to the attributes and the operation interface change track;
for example, the user firstly activates the operation interface of the word program into a current active window by clicking the word. Exe icon of the task bar through the attribute of the application software and the change track of the operation interface, and then views the document or edits the file; after a while, the user activates the micro-letter operation interface by clicking the micro-letter icon of the system tray to perform instant messaging (e.g. send message). Through the switching of the application software and the change of the operation interface, the real-time tracking of the behavior of the user can be realized.
Step S50: and carrying out audit analysis on the operation behaviors.
It should be noted that, in order to effectively implement audit analysis on the operation behavior of the user, step S50 specifically includes:
step S501: acquiring the switching condition of application software within the preset time range of the target terminal;
the preset time may be 12 hours, one day, three days, or the like, and one day is used as the preset time in this embodiment.
It can be understood that the analysis is performed according to the switching condition of the application software, so that the user can know which application software is used by one day.
Step S502: counting the use duration of each application software in the preset time range according to the switching condition;
for example, 10 applications are used in the target terminal in one day, and the duration of each application used and the time period of the day in which the application is used can be counted by analyzing according to the switching condition of the applications.
Step S503: acquiring and counting the specific business behaviors of each application software in the preset time range;
for example, during the day, the remote operation and maintenance tool remotely logs in to which servers (which can be obtained by identifying the content of the login interface) the server addresses are respectively; such as how many OA systems the browser has accessed together, what the corresponding web sites are, respectively, and so on.
Step S504: and carrying out audit analysis according to the switching condition, the using time and the specific business behavior.
It can be understood that, by performing audit analysis on the switching condition, the using time length and the specific service behavior of the application software, it can be known that the user performs steps, duration of operation behavior, efficiency of task operation and whether there is a sensitive operation in the target terminal, for example: tampering, leakage and stealing of service data; access to illegal websites at servers, misoperations, messy operations on important servers, etc.
The embodiment captures a plurality of operation interfaces of application software of the target terminal; then sequentially carrying out window segmentation on the operation interface to obtain window information of each window in the operation interface; determining the attribute of the application software of the operation interface according to the window information, and determining the change track of the operation interface based on the window information of each operation interface; then, according to the attribute and the change track of the operation interface, tracking the operation behavior executed by the user; and finally, performing audit analysis on the operation behavior. According to the method, window segmentation is carried out on the multiple grabbed operation interfaces, so that window information of each window in the operation interfaces is obtained; and determining the attribute of the application software of the operation interface according to the window information, and tracking the operation behavior executed by the user according to the attribute and the change track of the operation interface.
Referring to fig. 3, fig. 3 is a flowchart illustrating a second embodiment of the software operation behavior auditing method according to the present invention.
Based on the first embodiment, in this embodiment, the step S30 specifically includes:
step S30': based on a preset application software matching algorithm, determining the attribute of the application software of the operation interface according to the window title, the window text, the window control, the window icon and the window layout, and determining the change track of the operation interface based on the window information of each operation interface.
In specific implementation, based on a preset application software matching algorithm, similarity matching is performed according to window titles, window characters, window controls, window icons and window layouts and window titles, window characters, window controls, window icons and window layouts of known application software, so that application software with the maximum similarity is obtained and used as an output result. The name of the application software and the version of the current application software can be obtained through a preset application software matching algorithm, and then the change track of the operation interface can be determined based on the window title, the window text, the window control, the window icon and the window layout of each operation interface on the basis of obtaining the attribute of the application software.
In order to effectively track the operation behavior of the user, the step S40 includes:
step S401: presuming the operation type currently executed by the user according to the attribute of the application software;
it will be appreciated that, after obtaining the attribute of the software to which the operation interface belongs, it is known to some extent what type of operation is currently being performed by the user. For example, if the current operator interface is word. Exe, then the user should be looking at the document or editing text.
Step S402: according to the change of window characters and window layout in the operation interface, tracking and calculating an operation sequence and operation contents of a user based on the operation type and a preset tracking algorithm;
it should be noted that, the preset tracking algorithm is a series of clear instructions for tracking the user operation behavior through the change of window characters and window layout. The operation sequence of the user is the sequence of specific operations performed by the user, for example: the user activates an operation interface of the word program into a current active window by clicking a word.exe icon of the taskbar, and then views a document or edits the document; after a while, the user activates the micro-letter operation interface by clicking the micro-letter icon of the system tray to perform instant messaging (e.g. send message). Through the changes of the operation interfaces, the operation sequence of the user can be known, and as described above, the user firstly operates a word program and then operates a WeChat … …
In a specific implementation, the change of the window text content and the window layout can be tracked through a preset tracking algorithm, so that the detailed operation behavior of the user is tracked. For example, there are "Mei Xi" 2 words on the operation input window in which the content identified by the nth operation interface is word, and "Mei Xi" several words that win 2022 world cup "on the operation input window in which the content identified by the n+1th operation interface is word, then it can be inferred that the user is inputting text. Then, the text background of the word "Mei Xi wins the world cup of 2022" identified in the n+2th operation interface appears a short shade, the text input box of the word "Mei Xi wins the world cup of 2022" identified in the n+3 th operation interface appears a little letter (the words appear in a short time), and then it can be inferred that the user copies the file on the word to the little letter input box for message transmission. Thus, the operation sequence and the operation content of the tracking calculation user are realized.
Step S403: and recording operation behaviors executed by the user according to the operation sequence and the operation content.
Tracking and calculating an operation sequence and operation contents of a user according to the preset tracking algorithm; and then according to the user operation sequence and the corresponding operation content, the operation behavior executed by the user can be determined, and the operation behavior is converted into text information to be recorded.
For example, after determining the operation behavior of the user, the operation behavior is converted into text information, for example: the user starts to operate the word input box at 10 minutes 0 seconds at 2023, month 1, 3 for 10 minutes, the associated input window text is "Mei Xi wins 2022 world cup … …", then starts to operate the WeChat input box at 10 minutes 0 seconds at 2023, month 1, 3 for 2 minutes, the associated input text is "ABCDEFGHIJK … …".
The embodiment captures a plurality of operation interfaces of application software of the target terminal; then sequentially carrying out window segmentation on the operation interface to obtain window information of each window in the operation interface; determining the attribute of the application software of the operation interface according to the window information, and determining the change track of the operation interface based on the window information of each operation interface; then, the type of the operation currently executed by the user is presumed according to the attribute of the application software, the operation sequence and the operation content of the user are tracked and calculated according to the change of window characters and window layout in the operation interface and based on the operation type and a preset tracking algorithm, and the operation behavior executed by the user is reproduced according to the operation sequence and the operation content; and finally, performing audit analysis on the operation behavior. Because the invention presumes the operation type that the user carries out at present according to the attribute of the said application software, according to the change of window characters and window layout in the said operation interface, and based on said operation type and preset tracking algorithm, trace and calculate and get user's operation sequence and operation content, record user's operation behavior according to said operation sequence and said operation content, compared with the prior art, the invention has not merely solved the technical problem that the audit of the operation behavior of the application software is limited by operation system type in the prior art, also through tracing the change of window switching, window character content, have realized the record to user's operation behavior.
Referring to fig. 4, fig. 4 is a flowchart of a third embodiment of the software operation behavior auditing method according to the present invention.
Based on the above embodiments, in this embodiment, after step S40, the method further includes:
step S411: acquiring the system time and the corresponding operation type of the user operation sequence according to the operation behavior;
it should be noted that the operation type may be editing a document, browsing a web page, viewing a video, remotely logging in a server, chat with a WeChat, and the like.
Step S412: according to the operation type and the operation sequence, an operation interface before application software switching in the operation interface and an operation interface after application software switching in the operation interface are obtained and used as window switching screenshot;
it should be noted that, the first operation interface screenshot and the last operation interface screenshot before the application software in the operation interface is switched are saved in the database together (for example, when the operation interface is switched to the word operation interface, the operation interface screenshot of the word is switched to the WeChat from the word, and the last operation interface screenshot of the word is switched to the WeChat), so that the accuracy of the audit behavior can be ensured, and the subsequent audit tracing is convenient.
Step S413: and storing the system time, the operation type, the operation content and the window switching screenshot into a database.
It can be understood that the system time, the operation type, the operation content and the window screenshot are saved in the database, and when the operation behavior of a certain computer needs to be checked later, the specific operations executed by the user can be searched according to the time, the type and the keywords, and the screenshot of window switching can provide detailed information so as to ensure the accuracy of the auditing behavior.
Further, when the operation behavior audit analysis is performed, the user corresponding to the target terminal can be easily inquired about which operation is performed in which time period, how much time is continued, how efficiently the task operation is … …, and because text information is stored in the database instead of the traditional screen video stream, the storage space can be greatly saved, and secondly, keywords can be directly inquired when the operation behavior audit analysis is performed afterwards, without manually checking the video stream, the time and the labor are saved.
The embodiment captures a plurality of operation interfaces of application software of the target terminal; then sequentially carrying out window segmentation on the operation interface to obtain window information of each window in the operation interface; determining the attribute of the application software of the operation interface according to the window information, and determining the change track of the operation interface based on the window information of each operation interface; then, the type of the operation currently executed by the user is presumed according to the attribute of the application software, the operation sequence and the operation content of the user are tracked and calculated according to the change of window characters and window layout in the operation interface and based on the operation type and a preset tracking algorithm, and the operation behavior executed by the user is reproduced according to the operation sequence and the operation content; acquiring the system time and the corresponding operation type of the user operation sequence according to the operation behavior; according to the operation type and the operation sequence, an operation interface before application software switching in the operation interface and an operation interface after application software switching in the operation interface are obtained and used as window switching screenshot; storing the system time, the operation type, the operation content and the window switching screenshot into a database; and finally, performing audit analysis on the operation behavior. The invention converts the operation behavior into the text information and stores the text information in the database, and stores the first operation interface screenshot and the last operation interface screenshot before the application software is switched in the operation interface in the database instead of the traditional screen video stream, thereby greatly saving the storage space, ensuring the accuracy of the audit behavior, directly inquiring the keywords when the audit behavior is traced afterwards, and avoiding manual viewing of the video stream, and saving time and labor.
In addition, the embodiment of the invention also provides a storage medium, wherein the storage medium is stored with a software operation behavior auditing program, and the software operation behavior auditing program realizes the steps of the software operation behavior auditing method when being executed by a processor.
Referring to FIG. 5, FIG. 5 is a block diagram of a first embodiment of a software operational behaviour auditing system according to the present invention.
As shown in fig. 5, the software operation behavior auditing system provided by the embodiment of the present invention includes: an interface grabbing module 501, a window segmentation module 502, an information analysis module 503, an operation restoration module 504 and an audit analysis module 505.
The interface grabbing module 501 is configured to grab multiple operation interfaces of application software of a target terminal;
the window segmentation module 502 is configured to sequentially segment the operation interface to obtain window information of each window in the operation interface;
the information analysis module 503 is configured to determine an attribute of application software of the operation interface according to the window information, and determine a change track of the operation interface based on the window information of each operation interface;
the operation restoration module 504 is configured to track an operation behavior performed by a computing user according to the attribute and the operation interface change track;
the audit analysis module 505 is configured to perform audit analysis on the operation behavior;
the interface grabbing module 501 is further configured to grab a plurality of operation interfaces of the application software of the target terminal sequentially by installing audit software on the target terminal; or shooting the display screen of the target terminal at preset interval time or continuously shooting the display screen of the target terminal through the camera to obtain a plurality of operation interfaces of the application software of the target terminal; or, a conversion device for converting an analog signal into a digital signal is installed on a display signal line of the target terminal, so as to obtain a video display signal, and the video display signal is converted into a plurality of operation interfaces of application software of the target terminal.
The window segmentation module 502 is further configured to identify each window area according to RGB values of each operation interface, and obtain a window layout of the operation interface; sequentially carrying out window segmentation on each operation interface based on the window layout to obtain a plurality of windows; and identifying window titles, window characters, window icons and window controls in each window through an optical character identification technology and an image matching algorithm.
The audit analysis module 505 is further configured to obtain a switching condition of the application software within a preset time range of the target terminal; counting the use duration of each application software in the preset time range according to the switching condition; acquiring and counting the specific business behaviors of each application software in the preset time range; and carrying out audit analysis according to the switching condition, the using time and the specific business behavior.
The embodiment captures a plurality of operation interfaces of application software of the target terminal; then sequentially carrying out window segmentation on the operation interface to obtain window information of each window in the operation interface; determining the attribute of the application software of the operation interface according to the window information, and determining the change track of the operation interface based on the window information of each operation interface; then, according to the attribute and the change track of the operation interface, tracking the operation behavior executed by the user; and finally, performing audit analysis on the operation behavior. According to the method, window segmentation is carried out on the multiple grabbed operation interfaces, so that window information of each window in the operation interfaces is obtained; and determining the attribute of the application software of the operation interface according to the window information, and tracking the operation behavior executed by the user according to the attribute and the change track of the operation interface.
Based on the first embodiment of the software operation behavior auditing system of the present invention, a second embodiment of the software operation behavior auditing system of the present invention is provided.
In this embodiment, the information analysis module 503 is further configured to determine, based on a preset application matching algorithm, an attribute of the application of the operation interface according to a window title, a window text, a window control, a window icon, and a window layout.
The operation recovery module 504 is further configured to infer a type of operation currently performed by a user according to the attribute of the application software; according to the change of window characters and window layout in the operation interface, tracking and calculating an operation sequence and operation contents of a user based on the operation type and a preset tracking algorithm; and recording operation behaviors executed by the user according to the operation sequence and the operation content.
Other embodiments or specific implementation manners of the software operation behavior auditing system of the present invention may refer to the above method embodiments, and are not described herein.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of embodiments, it will be clear to a person skilled in the art that the above embodiment method may be implemented by means of software plus a necessary general hardware platform, but may of course also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. read-only memory/random-access memory, magnetic disk, optical disk), comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (9)
1. A method for auditing the operation behavior of software, the method comprising the steps of:
capturing a plurality of operation interfaces of application software of a target terminal;
sequentially carrying out window segmentation on the operation interface to obtain window information of each window in the operation interface, wherein the window information comprises window titles, window characters, window controls, window icons and window layouts;
determining the attribute of the application software of the operation interface according to the window information, and determining the change track of the operation interface based on the window information of each operation interface, wherein the change track of the operation interface comprises the change of window characters and window layout in the operation interface;
tracking operation behaviors executed by a computing user according to the attributes and the operation interface change track;
performing audit analysis on the operation behaviors;
the step of tracking the operation behavior executed by the computing user according to the attribute and the operation interface change track comprises the following steps:
the operation type currently executed by the user is presumed according to the attribute;
according to the change of window characters and window layout in the operation interface, tracking and calculating an operation sequence and operation contents of a user based on the operation type and a preset tracking algorithm;
and recording operation behaviors executed by the user according to the operation sequence and the operation content.
2. The method according to claim 1, wherein the step of determining the attribute of the application software of the operation interface according to the window information specifically includes:
and determining the attribute of the application software of the operation interface according to the window title, the window text, the window control, the window icon and the window layout based on a preset application software matching algorithm.
3. The method of claim 1, wherein the step of grabbing a plurality of operation interfaces of the application software of the target terminal comprises:
sequentially capturing a plurality of operation interfaces of application software of the target terminal by installing audit software on the target terminal;
or alternatively, the first and second heat exchangers may be,
shooting or continuously shooting a display screen of a target terminal at preset intervals by a camera to obtain a plurality of operation interfaces of application software of the target terminal;
or alternatively, the first and second heat exchangers may be,
a conversion device for converting an analog signal into a digital signal is installed on a display signal line of a target terminal, a video display signal is obtained, and the video display signal is converted into a plurality of operation interfaces of application software of the target terminal.
4. The method of claim 1, wherein following the step of tracking the operational behavior performed by the computing user based on the attributes and the operator interface change trajectory further comprises:
acquiring the system time and the corresponding operation type of the user operation sequence according to the operation behavior;
according to the operation type and the operation sequence, an operation interface before application software switching in the operation interface and an operation interface after application software switching in the operation interface are obtained and used as window switching screenshot;
and storing the system time, the operation type, the operation content and the window switching screenshot into a database.
5. The method of claim 2, wherein the step of sequentially performing window segmentation on the operation interface to obtain window information of each window in the operation interface includes:
identifying each window area according to the RGB value of each operation interface, and obtaining the window layout of the operation interface;
sequentially carrying out window segmentation on each operation interface based on the window layout to obtain a plurality of windows;
and identifying window titles, window characters, window icons and window controls in each window through an optical character identification technology and an image matching algorithm.
6. The method of claim 4, wherein the attributes of the application software include type, purpose, name and version of the application software, and the step of performing audit analysis on the operation behavior specifically includes:
acquiring the switching condition of application software within the preset time range of the target terminal;
counting the use duration of each application software in the preset time range according to the switching condition;
acquiring and counting the specific business behaviors of each application software in the preset time range;
and carrying out audit analysis according to the switching condition, the using time and the specific business behavior.
7. A software operational behaviour auditing system, the system comprising:
the interface grabbing module is used for grabbing a plurality of operation interfaces of the application software of the target terminal;
the window segmentation module is used for sequentially carrying out window segmentation on the operation interface to obtain window information of each window in the operation interface, wherein the window information comprises window titles, window characters, window controls, window icons and window layouts;
the information analysis module is used for determining the attribute of the application software of the operation interface according to the window information, and determining the change track of the operation interface based on the window information of each operation interface, wherein the change track of the operation interface comprises the change of window characters and window layout in the operation interface;
the operation restoration module is used for tracking the operation behavior executed by the user according to the attribute and the operation interface change track;
the audit analysis module is used for carrying out audit analysis on the operation behaviors;
the operation recovery module is also used for predicting the type of operation currently executed by the user according to the attribute; according to the change of window characters and window layout in the operation interface, tracking and calculating an operation sequence and operation contents of a user based on the operation type and a preset tracking algorithm; and recording operation behaviors executed by the user according to the operation sequence and the operation content.
8. A software operational behaviour auditing apparatus, the apparatus comprising: a memory, a processor, and a software operational behaviour auditing program stored on the memory and executable on the processor, the software operational behaviour auditing program configured to implement the steps of the software operational behaviour auditing method of any of claims 1 to 6.
9. A storage medium having stored thereon a software operational behaviour auditing program, which when executed by a processor implements the steps of the software operational behaviour auditing method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310181296.0A CN115859278B (en) | 2023-03-01 | 2023-03-01 | Method, system, equipment and storage medium for auditing software operation behaviors |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310181296.0A CN115859278B (en) | 2023-03-01 | 2023-03-01 | Method, system, equipment and storage medium for auditing software operation behaviors |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115859278A CN115859278A (en) | 2023-03-28 |
| CN115859278B true CN115859278B (en) | 2023-05-23 |
Family
ID=85659430
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310181296.0A Active CN115859278B (en) | 2023-03-01 | 2023-03-01 | Method, system, equipment and storage medium for auditing software operation behaviors |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115859278B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109656704A (en) * | 2018-12-21 | 2019-04-19 | 北京天融信网络安全技术有限公司 | A kind of information processing method and information processing unit |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104700231A (en) * | 2015-03-27 | 2015-06-10 | 福州大学 | Desktop screen auditing method based on user behaviors |
| CN107273145A (en) * | 2016-04-06 | 2017-10-20 | 中兴通讯股份有限公司 | A kind of method of managing software and device, terminal |
| CN108229481B (en) * | 2017-12-25 | 2020-09-11 | ***通信集团江苏有限公司 | Screen content analysis method and device, computing equipment and storage medium |
| CN113342622A (en) * | 2021-08-02 | 2021-09-03 | 深圳市永达电子信息股份有限公司 | Operation behavior auditing method and device and storage medium |
| CN114187658A (en) * | 2021-12-01 | 2022-03-15 | 上海上讯信息技术股份有限公司 | Input operation identification method and equipment |
-
2023
- 2023-03-01 CN CN202310181296.0A patent/CN115859278B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109656704A (en) * | 2018-12-21 | 2019-04-19 | 北京天融信网络安全技术有限公司 | A kind of information processing method and information processing unit |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115859278A (en) | 2023-03-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9998598B1 (en) | Methods and systems for automatically recognizing actions in a call center environment using screen capture technology | |
| JP6207185B2 (en) | Information analysis apparatus, information analysis method, information analysis system, and program | |
| US20050210056A1 (en) | Workstation information-flow capture and characterization for auditing and data mining | |
| US20110038542A1 (en) | Computer application analysis | |
| CN103530218A (en) | Monitoring triggering method based on behavior detection | |
| CN111324440A (en) | Method, device and equipment for executing automation process and readable storage medium | |
| CN110798440B (en) | Abnormal user detection method, device and system and computer storage medium | |
| CN111221625B (en) | File detection method, device and equipment | |
| CN111552633A (en) | Interface abnormal call testing method and device, computer equipment and storage medium | |
| Khan et al. | Digital forensics and cyber forensics investigation: security challenges, limitations, open issues, and future direction | |
| CN114692049A (en) | Browser-based screen recording method and device, electronic equipment and storage medium | |
| CN112307464A (en) | Fraud identification method and device and electronic equipment | |
| CN112416212B (en) | Program access method, apparatus, electronic device and readable storage medium | |
| US10038785B1 (en) | Methods and systems for automatically recognizing actions in a call center environment using video data | |
| CN116432210B (en) | File management method and system based on security protection | |
| CN112905935A (en) | Page recording method, page recording animation generation method, equipment and storage medium | |
| CN115859278B (en) | Method, system, equipment and storage medium for auditing software operation behaviors | |
| EP3262509B1 (en) | Remote supervision of client device activity | |
| US20090158090A1 (en) | Data entry retrieval | |
| CN104158696A (en) | Determination method and device for measuring delayed operation time and terminal | |
| CN115858049A (en) | RPA flow componentization arrangement method, device, equipment and medium | |
| JP6501159B2 (en) | Analysis and translation of operation records of computer devices, output of information for audit and trend analysis device of the system. | |
| CN113765924A (en) | Safety monitoring method, terminal and equipment based on cross-server access of user | |
| Krieter et al. | Track every move of your students: Log files for learning analytics from mobile screen recordings | |
| CN112000559A (en) | Abnormal equipment detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |