CN110569416B - APP control processing method based on data crawling and related products - Google Patents
APP control processing method based on data crawling and related products Download PDFInfo
- Publication number
- CN110569416B CN110569416B CN201910832368.7A CN201910832368A CN110569416B CN 110569416 B CN110569416 B CN 110569416B CN 201910832368 A CN201910832368 A CN 201910832368A CN 110569416 B CN110569416 B CN 110569416B
- Authority
- CN
- China
- Prior art keywords
- data
- information
- app
- information block
- different
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The embodiment of the invention discloses a processing method of an APP control based on data crawling and a related product, comprising the following steps: acquiring data respectively corresponding to element attributes of different controls of a first information block in an APP interface; acquiring data respectively corresponding to the element attributes of different controls of other information blocks in the APP interface according to the data respectively corresponding to the element attributes of different controls of the first information block; receiving a data crawling request sent by a terminal, wherein the data crawling request carries first information and is used for indicating to acquire an APP interface matched with the first information; and adjusting the arrangement of the different information blocks according to the different data of the different information blocks to obtain an APP interface matched with the first information. By adopting the means, the data of other information blocks can be acquired according to the data information of the first information block, so that the data crawling method is very convenient, the crawling efficiency is higher, the operation difficulty is lower, and the technical requirement of data crawling is greatly reduced.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a processing method of an APP control based on data crawling and a related product.
Background
Data crawling: the method is used for automatically acquiring network data on a large scale through a certain rule.
Display layer: i.e., UI layer, refers to the portion of the APP architecture that presents data to the user. Including but not limited to interface layout, screen fit, text size, color, picture resources, prompts, animations, etc.
APP control: APP controls refer to technical functions in APP development to accomplish a specific presentation or specific page/form. Such as buttons, a pop-up window, an input box, etc.
Control element attributes: the control element refers to attribute information such as an ID, a text name, a type, a size, whether the APP control is operable or not and the like.
The prior art generally uses Web crawlers to obtain interface data, where the data embedded in the structure and style of a website is obtained by traversing the access website. One is to crawl text information from web site HTML code; the other is to request a link to acquire the json file through a website. However, the Web crawler method is only suitable for crawling data on websites, the websites have respective limit regulations and anticreeper strategies, and crawling website data may be banned by IP, and even subject to legal litigation.
In the prior art, an API crawler method of a mobile phone APP is adopted, firstly, network requests sent and received by the APP are simulated through a network proxy tool, thereby capturing an API interface containing a required data packet in the network request process, then, parameters of the API interface are changed, and the requests are sent for multiple times to obtain more data. However, as the android system has a mechanism for limiting proxy access, the identified proxy request cannot return data, and the API crawler of the mobile phone APP has high requirements on skills, and needs to be familiar with a network transmission protocol and to be skilled in applying a network proxy tool; meanwhile, the time consumption is long, one action can return to a plurality of interfaces, and the process of searching and verifying the interfaces needs a long time; and most of APP can carry out private key encryption on data, and no private key is difficult to decrypt the crawled data, so the data decryption difficulty is high.
Disclosure of Invention
The embodiment of the application provides a processing method of an APP control based on data crawling and a related product, which can improve the data crawling efficiency and reduce the operation difficulty.
A first aspect of an embodiment of the present application provides a method for processing an APP control based on data crawling, including:
acquiring data respectively corresponding to element attributes of different controls of a first information block in an APP interface;
Acquiring data respectively corresponding to the element attributes of different controls of other information blocks in the APP interface according to the data respectively corresponding to the element attributes of different controls of the first information block;
receiving a data crawling request sent by a terminal, wherein the data crawling request carries first information and is used for indicating to acquire an APP interface matched with the first information;
and adjusting the arrangement of the different information blocks according to the different data of the different information blocks to obtain an APP interface matched with the first information.
Optionally, the first information block includes at least a first control, and the obtaining data corresponding to element attributes of different controls of the first information block in the APP application interface includes:
acquiring a display layer corresponding to the APP interface;
acquiring data corresponding to a first information block in the display layer and the position of the data of the first information block in the display layer;
and acquiring data corresponding to the element attribute of the first control from the data of the first information block, and acquiring the position of the element attribute of the first control in the data of the first information block.
Optionally, the APP application interface further includes at least a second information block, where the second information block includes at least a second control, and the obtaining, according to data corresponding to element attributes of different controls of the first information block, data corresponding to element attributes of different controls of other information blocks in the APP application interface includes:
acquiring data of the second information block according to the position of the data of the first information block in the display layer;
and acquiring data corresponding to the element attribute of the second control from the data of the second information block according to the position of the element attribute of the first control in the data of the first information block.
Optionally, after the obtaining the data corresponding to the element attributes of the different controls of the other information blocks in the APP application interface according to the data corresponding to the element attributes of the different controls of the first information block, before the adjusting the arrangement of the different information blocks according to the different data of the different information blocks to obtain the APP application interface matched with the first information, the method includes:
confirming whether all information blocks in the APP interface are acquired;
And if all the information blocks in the APP interface are acquired, triggering and executing to adjust the arrangement of the different information blocks according to the different data of the different information blocks so as to obtain the APP interface matched with the first information.
Optionally, the determining whether all the information blocks in the APP application interface are acquired includes:
confirming whether a display layer corresponding to the APP interface has an end mark or not;
if yes, confirming that all information blocks in the APP interface are acquired according to the position of the ending mark.
Optionally, if the display layer corresponding to the APP application interface does not have an end mark, the determining whether all the information blocks in the APP application interface are acquired completely includes:
acquiring first data in a display layer corresponding to the APP interface;
executing a screen sliding program to acquire second data in a display layer corresponding to the APP interface;
confirming whether the first data and the second data are consistent;
and if the information blocks are consistent, confirming that all the information blocks in the APP interface are acquired.
Optionally, the adjusting the arrangement of the different information blocks according to the different data of the different information blocks to obtain an APP application interface matched with the first information includes:
Respectively acquiring data related to the first information in different information blocks;
sorting the data according to the order from high to low to obtain sorted data;
acquiring information blocks respectively corresponding to the sequenced data;
and displaying the information blocks respectively corresponding to the sequenced data to an APP interface.
A second aspect of an embodiment of the present application provides a processing system for an APP control based on data crawling, including:
the first acquisition module is used for acquiring data corresponding to element attributes of different controls of the first information block in the APP interface;
the second acquisition module is used for acquiring the data respectively corresponding to the element attributes of the different controls of the other information blocks in the APP application interface according to the data respectively corresponding to the element attributes of the different controls of the first information block;
the data crawling module is used for receiving a data crawling request sent by a terminal, wherein the data crawling request carries first information and is used for indicating to acquire an APP interface matched with the first information;
and the data processing module is used for adjusting the arrangement of the different information blocks according to the different data of the different information blocks so as to obtain an APP interface matched with the first information.
A third aspect of the embodiments of the present application provides a server, including a processor, an input device, an output device, and a memory, where the processor, the input device, the output device, and the memory are connected to each other, and where the memory is configured to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to perform the method.
A fourth aspect of the embodiments of the present application provides a computer-readable storage medium storing a computer program for execution by a processor to implement the method.
The embodiment of the application has at least the following beneficial effects:
according to the embodiment of the application, the data respectively corresponding to the element attributes of the different controls of the other information blocks in the APP interface are obtained according to the data respectively corresponding to the element attributes of the different controls of the first information block by obtaining the data respectively corresponding to the element attributes of the different controls of the first information block in the APP interface; and then adjusting the arrangement of the different information blocks to obtain an APP interface matched with the user requirements. By adopting the means, the data of other information blocks can be acquired according to the data information of the first information block, so that the data crawling method is very convenient, the crawling efficiency is higher, the operation difficulty is lower, and the technical requirement of data crawling is greatly reduced.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Wherein:
fig. 1 is a schematic diagram of a network architecture according to an embodiment of the present invention;
fig. 2 is a schematic view of a scenario of a processing method of an APP control based on data crawling provided by an embodiment of the present invention;
fig. 3 is a flow diagram of a processing method of an APP control based on data crawling according to an embodiment of the present invention;
fig. 4 is a flow diagram of a processing method of an APP control based on data crawling according to an embodiment of the present invention;
fig. 5a is a schematic diagram of a screen sliding termination judgment of an APP control based on data crawling according to an embodiment of the present invention;
FIG. 5b is another schematic diagram of a determination of end of sliding screen of an APP control based on data crawling according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a processing method of an APP control based on data crawling according to an embodiment of the present invention;
Fig. 7 is a schematic structural diagram of a server according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of a processing system of an APP control based on data crawling according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
The terms first, second and the like in the description and in the claims and in the above-described figures are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the described embodiments of the application may be combined with other embodiments.
Referring to fig. 1, fig. 1 is a schematic diagram of a network architecture according to an embodiment of the present application. The network architecture may include a plurality of servers and a plurality of terminal devices, as shown in fig. 1, specifically includes a terminal device 100a, a terminal device 100b, a terminal device 100c, and a server 200a, where the server 200a may perform data transmission with each terminal device through a network, each terminal device may install a different APP application, and the server 200a may be a processing server based on an APP control that is crawled by data, so that each terminal device may perform data transmission with the server 200a, for example, the server 200a may send update information of an APP application interface after data crawling to each terminal device. The terminal device may include a cell phone, tablet, notebook, palm top, mobile internet device (mobile internet device, MID), wearable device (e.g., smart watch, smart bracelet, etc.). Each terminal device can receive the update information of the APP application interface after the data crawling sent by the server 200 a.
The update information of the APP application interface displayed in each terminal device may be different, and specific update information thereof may be determined by a user request corresponding to the terminal device, and the update information of the APP application interface corresponding to the user request is different according to different user requests. Referring to fig. 2, a schematic view of a scenario of a processing method of an APP control based on data crawling according to an embodiment of the present application is shown. As shown in fig. 2, taking the terminal device 100a in the embodiment corresponding to fig. 1 as an example, the server 200 may include the server 200a in the embodiment corresponding to fig. 1, after the terminal device 100a opens the face-to-face application interface, first, a default first page of the face-to-face application may be displayed in the terminal display interface, in the first page, several function options may be displayed, such as a "delicious" option, "a" nearby "option and a" smart sorting "option, respectively, when the user selects the" delicious "option 400, the terminal device 100a may jump to a presentation page corresponding to the" delicious "option 400, and when the user selects the" smart sorting "option 401, the terminal device 100a may send a request to the server 200 in response to a selection operation of the user for the" smart sorting "option 401, so as to request to obtain updated information of the APP application interface after data crawling. The server 200 may obtain the data information of the current application interface according to the request, then sort the data information according to the information corresponding to the request, and further obtain the update information of the APP application interface, and then the server 200 sends the update information to the terminal device 100a for display. Specifically, the server 200 firstly obtains, from the presentation layer database, data corresponding to element attributes of different controls of the first information block 210 of the beauty group application interface in the terminal device 100a, where the element attributes of the different controls include a shop name, a month average person number, a location, and preference information, respectively. The display layer mainly refers to an interface for interaction with a user, and is used for receiving data input by the user and displaying the data required by the user after processing. The server 200 obtains a display layer corresponding to the APP interface; then, acquiring data corresponding to a first information block in the display layer and the position of the data of the first information block in the display layer; and acquiring data corresponding to the element attribute of the first control from the data of the first information block, and acquiring the position of the element attribute of the first control in the data of the first information block. Then acquiring data of other information blocks, such as a second information block and a third information block, wherein the data of the second information block and the third information block are acquired according to the position of the data of the first information block in the display layer; and acquiring the data corresponding to the element attribute of the second control from the data of the second information block according to the position of the element attribute of the first control in the data of the first information block, and acquiring the data corresponding to the element attribute of the third control from the data of the third information block. And then, based on the intelligent ordering request sent by the terminal device 100a, ordering the acquired data according to the ordering request to obtain an updated application interface. The scheme can be used for acquiring public data such as merchant data, electronic commerce commodity data, platform resident user data and the like on the APP online, and can further use the data to perform data analysis of various scenes such as bid analysis, sales evaluation, content classification, regional distribution and the like. The above is merely a method of processing data, and is not particularly limited. In a typical application scenario, such as acquisition of online merchant data of group purchase APP, the scheme can acquire data of shop names, addresses, contact ways, average consumption, popularity, preferential activities, online services, online payment ways, user evaluation, recommended commodities and the like of merchants in the APP. Further, these data may be used for various analyses such as sales assessment, pay-off on-line service layout, regional distribution, etc. According to the embodiment of the application, the data respectively corresponding to the element attributes of the different controls of the other information blocks in the APP interface are obtained according to the data respectively corresponding to the element attributes of the different controls of the first information block by obtaining the data respectively corresponding to the element attributes of the different controls of the first information block in the APP interface; and then adjusting the arrangement of the different information blocks to obtain an APP interface matched with the user requirements. By adopting the means, the data of other information blocks can be acquired according to the data information of the first information block, so that the data crawling method is very convenient, the crawling efficiency is higher, the operation difficulty is lower, and the technical requirement of data crawling is greatly reduced.
The storage data related to the scheme can be stored into a blockchain, such as the display layer data, which can be placed in the blockchain; the data of the APP application interface may also be stored in a blockchain, etc. The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. The Blockchain (Blockchain), which is essentially a decentralised database, is a string of data blocks that are generated by cryptographic means in association, each data block containing a batch of information of network transactions for verifying the validity of the information (anti-counterfeiting) and generating the next block. The blockchain may include a blockchain underlying platform, a platform product services layer, and an application services layer.
The blockchain underlying platform may include processing modules for user management, basic services, smart contracts, operation monitoring, and the like. The user management module is responsible for identity information management of all blockchain participants, including maintenance of public and private key generation (account management), key management, maintenance of corresponding relation between the real identity of the user and the blockchain address (authority management) and the like, and under the condition of authorization, supervision and audit of transaction conditions of certain real identities, and provision of rule configuration (wind control audit) of risk control; the basic service module is deployed on all block chain node devices, is used for verifying the validity of a service request, recording the service request on a storage after the effective request is identified, for a new service request, the basic service firstly analyzes interface adaptation and authenticates the interface adaptation, encrypts service information (identification management) through an identification algorithm, and transmits the encrypted service information to a shared account book (network communication) in a complete and consistent manner, and records and stores the service information; the intelligent contract module is responsible for registering and issuing contracts, triggering contracts and executing contracts, a developer can define contract logic through a certain programming language, issue the contract logic to a blockchain (contract registering), invoke keys or other event triggering execution according to the logic of contract clauses to complete the contract logic, and simultaneously provide a function of registering contract upgrading; the operation monitoring module is mainly responsible for deployment in the product release process, modification of configuration, contract setting, cloud adaptation and visual output of real-time states in product operation, for example: alarms, monitoring network conditions, monitoring node device health status, etc.
The platform product service layer provides basic capabilities and implementation frameworks of typical applications, and developers can complete the blockchain implementation of business logic based on the basic capabilities and the characteristics of the superposition business. The application service layer provides the application service based on the block chain scheme to the business participants for use.
Specifically, if the server obtains the data corresponding to the element attributes of the different controls of the first information block in the APP application interface, the server sends a request to the blockchain, the blockchain can evaluate the request, whether the data is authorized to be obtained or not is confirmed by obtaining the corresponding identification information of the server, if so, the blockchain obtains the information by inquiring the corresponding node information storing the data, and sends the information to the server. The server obtains data corresponding to the element attributes of different controls of other information blocks in the APP interface according to the data corresponding to the element attributes of different controls of the first information block, and then the data can be stored in a preset blockchain. When a server receives a data crawling request sent by a terminal, the server acquires data corresponding to the APP interface from a blockchain, and then adjusts the arrangement of different information blocks according to the different data of the different information blocks to obtain the APP interface matched with the first information. By adopting the means, the data management is more orderly, the data risk is reduced, and the data security is improved.
Referring to fig. 3, fig. 3 is a flow chart of a processing method of an APP control based on data crawling according to an embodiment of the present invention. As shown in fig. 3, the method includes steps 301-304, specifically as follows:
301. the method comprises the steps that a server obtains data respectively corresponding to element attributes of different controls of a first information block in an APP interface;
the APP application may be any application. Optionally, the method can probe the APP control and the elements thereof under the android automation test framework uiautomatic, and automatically operate a terminal such as a mobile phone by triggering the control to simulate the action of browsing the APP by a user, and simultaneously acquire text data in the control elements of the APP display layer. Before data acquisition, the terminal and the server are required to be connected, for example, the terminal and the server are divided into two types of network connection and USB line connection. The network connection is that the server, such as the PC end, is connected with the automatic testing tool in the PC end through the IP address of the mobile end network under the condition that the PC end and the mobile end mobile phone are in the same local area network. The other is USB line connection, which is to connect the PC end and the mobile end through a USB data line, but the mobile end is required to start the function of 'USB debugging', and the connected PC end is allowed to debug the mobile end; in this case, the automatic test tool in the PC is connected via the device_id of the mobile terminal DEVICE. After the APP is opened by the mobile terminal, a control checking function in the server-side automatic testing tool is adopted, and the control and the element thereof of the data display layer in the APP are identified.
The first information block may be an information block arranged in a list in the application interface and located at a first position. For example, all information of a first store in the beauty-group application forms a first information block, and the second information block and the third information block are sequentially shown as an example, which is not limited herein.
As shown in fig. 2, the element attribute of the different controls of the first information block may be a shop name of the first information block, or a month average person number of the first information block. Exemplary data for the first information block of fig. 2 is shown in table one.
List one
| ID | X | Y | Text |
| 1 | 25 | 30 | Shanzhi beef chafing dish |
| 2 | 25 | 40 | Average moon popularity 90 |
| 3 | 25 | 50 | South-head 90km |
| 4 | 25 | 60 | Every 100 yuan of the buying bill is reduced by 12 yuan |
Specifically, step 301 may include:
acquiring a display layer corresponding to the APP interface;
acquiring data corresponding to a first information block in the display layer and the position of the data of the first information block in the display layer;
the position may be the layer located on the display layer.
And acquiring data corresponding to the element attribute of the first control from the data of the first information block, and acquiring the position of the element attribute of the first control in the data of the first information block.
302. The server obtains data respectively corresponding to the element attributes of different controls of other information blocks in the APP interface according to the data respectively corresponding to the element attributes of different controls of the first information block;
the data corresponding to the element attributes of the different controls of the first information block can be obtained from a display layer corresponding to the APP interface. The display layer contains all data information of the APP interface, and the display layer can distribute the data information based on a tree structure. And if the first control in the first information block is a shop name and the hierarchy is a fifth layer in the first information block data, the hierarchy of the shop name in the second information block in the display layer is also the fifth layer. The foregoing is merely an example, and is not intended to be limiting in any way.
The APP application interface further includes at least a second information block, where the second information block includes at least a second control, and the obtaining, according to data corresponding to element attributes of different controls of the first information block, data corresponding to element attributes of different controls of other information blocks in the APP application interface includes:
Acquiring data of the second information block according to the position of the data of the first information block in the display layer;
and acquiring data corresponding to the element attribute of the second control from the data of the second information block according to the position of the element attribute of the first control in the data of the first information block.
And the like to obtain data corresponding to the element attributes of different controls of all the information blocks.
303. The server receives a data crawling request sent by a terminal, wherein the data crawling request carries first information and is used for indicating to acquire an APP interface matched with the first information;
the order of step 303 and step 301 may be adjusted, which is not limited herein. The server can automatically acquire the data information of the current interface when detecting the corresponding terminal application interface, and save the acquired data information so as to meet the user requirement.
Alternatively, the data information may be acquired after receiving the request from the user.
The first information may be an acquisition request for an element attribute of a control, or arrangement information of the element attribute of the control in the current interface. For example, the store with the highest popularity or the store closest to the store can be used.
304. And the server adjusts the arrangement of the different information blocks according to the different data of the different information blocks so as to obtain an APP application interface matched with the first information.
Wherein step 304 may comprise:
respectively acquiring data related to the first information in different information blocks;
sorting the data according to the order from high to low to obtain sorted data;
here, the order rule from high to low, from large to small, from small to large, etc. is not limited.
Acquiring information blocks respectively corresponding to the sequenced data;
and displaying the information blocks respectively corresponding to the sequenced data to an APP interface.
The above is not limited to obtaining the complete information after sorting, and only one of the complete information may be selected for display.
If the user wants to acquire the names of shops with the most preferential, acquiring preferential information data of all the information blocks from all the acquired data information, and sequencing the preferential information data corresponding to all the information blocks in the order from high to low or from high to low, so as to obtain the updated arrangement of the information blocks, namely obtaining an APP interface matched with the preferential information; alternatively, only the information of the first store with the greatest preference may be displayed.
According to the embodiment of the application, the data respectively corresponding to the element attributes of the different controls of the other information blocks in the APP interface are obtained according to the data respectively corresponding to the element attributes of the different controls of the first information block by obtaining the data respectively corresponding to the element attributes of the different controls of the first information block in the APP interface; and then adjusting the arrangement of the different information blocks to obtain an APP interface matched with the user requirements. By adopting the means, the data of other information blocks can be acquired according to the data information of the first information block, so that the data crawling method is very convenient, the crawling efficiency is higher, the operation difficulty is lower, and the technical requirement of data crawling is greatly reduced.
According to the scheme, the action of browsing the APP data by the user is simulated, all target data are traversed, and meanwhile, the data are crawled by acquiring the control elements of the APP data display layer. The scheme can avoid the data encryption and anti-crawler means in the existing crawling data process, and reduces the operation difficulty. According to the scheme, the control elements are used for directly crawling data from the APP data display layer, and the obtained data can be directly used; no spying API interface and decryption of the crawled data is required. The method effectively avoids legal risks and anticreeper strategies, and has no problem of crawler risks. The application simulates normal browsing behavior of a user and does not touch the data access limit of an APP developer. And breaks through the limitation of data encryption and decryption, directly crawls data from the APP display layer, and directly obtains unencrypted data. And API interface prying and data decryption operations are not required, and professional network transmission knowledge is not required to be mastered. On the other hand, it shortens the data crawling time. The data seen at the corresponding interface of the APP can be crawled from the corresponding interface, and the obtained data can be directly read; the crawling pre-grinding time and the data processing time are shortened compared with the API crawler.
Referring to fig. 4, fig. 4 is a flow chart of a processing method of an APP control based on data crawling according to an embodiment of the present application. As shown in fig. 4, it may include steps 401-405, which are specifically as follows:
401. acquiring data respectively corresponding to element attributes of different controls of a first information block in an APP interface;
the first information block may be an information block arranged in a list in the application interface and located at a first position. For example, all information of a first store in the beauty application forms a first information block, and the second information block and the third information block are sequentially arranged below the first information block
Acquiring a display layer corresponding to the APP interface; acquiring data corresponding to a first information block in the display layer and the position of the data of the first information block in the display layer; and acquiring data corresponding to the element attribute of the first control from the data of the first information block, and acquiring the position of the element attribute of the first control in the data of the first information block.
402. Acquiring data respectively corresponding to the element attributes of different controls of other information blocks in the APP interface according to the data respectively corresponding to the element attributes of different controls of the first information block;
Specifically, acquiring the data of the second information block according to the position of the data of the first information block in the display layer; and acquiring data corresponding to the element attribute of the second control from the data of the second information block according to the position of the element attribute of the first control in the data of the first information block. And the like to obtain data corresponding to the element attributes of different controls of all the information blocks
403. Receiving a data crawling request sent by a terminal, wherein the data crawling request carries first information and is used for indicating to acquire an APP interface matched with the first information;
the first information may be an acquisition request for an element attribute of a control, or arrangement information of the element attribute of the control in the current interface. For example, the store with the highest popularity or the store closest to the store can be used.
404. Confirming whether all information blocks in the APP interface are acquired;
the data in the APP is not generally like a page turning function in a website, but more data can be displayed by sliding downwards. The data in the APP can be displayed completely through the automatic sliding screen, so that the purpose of traversing the data is achieved, and the aim that all information blocks in the APP interface are acquired is achieved.
Fig. 5a and fig. 5b are schematic diagrams of a screen sliding termination judgment of an APP control based on data crawling according to an embodiment of the present invention. Optionally, after the data are completely displayed in the APP, the sliding screen program is terminated. And the server executes a screen sliding program to confirm that all the data in the APP are displayed.
The method for judging whether the data is displayed is as follows:
confirming whether a display layer corresponding to the APP interface has an end mark or not;
if yes, confirming that all information blocks in the APP interface are acquired according to the position of the ending mark.
After the display of part of APP data is completed, a mark for prompting a user that no more data exists is arranged on the display layer, and whether all data are displayed or not can be judged according to the mark. If the information block close to the ending mark is the last information block, and when the ending mark is acquired, the information blocks in the APP interface can be confirmed to be acquired.
For the APP display layer without the end mark, whether all data are displayed is judged whether to be completed by comparing whether the data before and after the sliding screen are repeated.
Specifically, if the display layer corresponding to the APP application interface does not have an end mark, the determining whether all the information blocks in the APP application interface are acquired includes:
Acquiring first data in a display layer corresponding to the APP interface;
executing a screen sliding program to acquire second data in a display layer corresponding to the APP interface;
confirming whether the first data and the second data are consistent;
and if the information blocks are consistent, confirming that all the information blocks in the APP interface are acquired.
If the data are inconsistent, continuing to execute the screen sliding program, and continuously judging until the position where the repeated data exist is obtained.
The data in the APP is carried through a control of the display layer, and each piece of text data is stored in a control element of the control of the display layer when the current page of the APP is displayed. And in the process of displaying data by sliding the screen, the element attribute of the control is synchronously called, and the text data in the element attribute is stored, so that the purposes of acquiring and storing APP information are achieved.
405. And if all the information blocks in the APP interface are acquired, adjusting the arrangement of the different information blocks according to the different data of the different information blocks to obtain the APP interface matched with the first information.
In this embodiment, the order of the steps 403 and 401-402 may be adjusted, and only the data of the element attribute of the control consistent with the information carried in the step 403 may be obtained, so as to further improve the efficiency of data crawling.
According to the scheme, the action of browsing the APP data by the user is simulated, all target data are traversed, and meanwhile, the data are crawled by acquiring the control elements of the APP data display layer. The scheme can avoid the data encryption and anti-crawler means in the existing crawling data process, and reduces the operation difficulty.
Furthermore, the scheme can also explore the processes of data display control and programming, is optimized as a visual operation tool, and automatically generates crawling codes according to user operation examples in the background, so that a data crawler is not required to have program coding capability, and a non-programmer can also crawl data by himself. Specifically, as shown in fig. 6, the basic operation is manually demonstrated based on the APP application interface, and the steps of manual operation are recorded in the background, and then the simulated operation code is generated according to the recording. Based on the demonstration, according to the appointed target data, the target data is crawled by sending out a starting instruction, and when the target data is completed, the target data is automatically stored and sent out.
In accordance with the foregoing embodiments, referring to fig. 7, fig. 7 is a schematic structural diagram of a server according to an embodiment of the present application, where the server includes a processor, an input device, an output device, and a memory, where the processor, the input device, the output device, and the memory are connected to each other, and the memory is configured to store a computer program, where the computer program includes program instructions, where the processor is configured to invoke the program instructions, and where the program includes instructions for performing the following steps;
Acquiring data respectively corresponding to element attributes of different controls of a first information block in an APP interface;
acquiring data respectively corresponding to the element attributes of different controls of other information blocks in the APP interface according to the data respectively corresponding to the element attributes of different controls of the first information block;
receiving a data crawling request sent by a terminal, wherein the data crawling request carries first information and is used for indicating to acquire an APP interface matched with the first information;
and adjusting the arrangement of the different information blocks according to the different data of the different information blocks to obtain an APP interface matched with the first information.
According to the embodiment of the application, the data respectively corresponding to the element attributes of the different controls of the other information blocks in the APP interface are obtained according to the data respectively corresponding to the element attributes of the different controls of the first information block by obtaining the data respectively corresponding to the element attributes of the different controls of the first information block in the APP interface; and then adjusting the arrangement of the different information blocks to obtain an APP interface matched with the user requirements. By adopting the means, the data of other information blocks can be acquired according to the data information of the first information block, so that the data crawling method is very convenient, the crawling efficiency is higher, the operation difficulty is lower, and the technical requirement of data crawling is greatly reduced.
The foregoing description of the embodiments of the present application has been presented primarily in terms of a method-side implementation. It will be appreciated that, in order to achieve the above-mentioned functions, the terminal includes corresponding hardware structures and/or software modules for performing the respective functions. Those of skill in the art will readily appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The embodiment of the application can divide the functional units of the terminal according to the method example, for example, each functional unit can be divided corresponding to each function, and two or more functions can be integrated in one processing unit. The integrated units may be implemented in hardware or in software functional units. It should be noted that, in the embodiment of the present application, the division of the units is schematic, which is merely a logic function division, and other division manners may be implemented in actual practice.
In accordance with the foregoing, referring to fig. 8, fig. 8 is a schematic structural diagram of a processing device of an APP control based on data crawling according to an embodiment of the present application. It includes a first acquisition module 801, a second acquisition module 802, a reception request module 803, and a data processing module 804, specifically:
the first obtaining module 801 is configured to obtain data corresponding to element attributes of different controls of the first information block in the APP application interface;
a second obtaining module 802, configured to obtain data corresponding to element attributes of different controls of other information blocks in the APP application interface according to data corresponding to element attributes of different controls of the first information block respectively;
a receiving request module 803, configured to receive a data crawling request sent by a terminal, where the data crawling request carries first information, and the data crawling request is used to instruct to obtain an APP application interface that is matched with the first information;
the data processing module 804 is configured to adjust the arrangement of the different information blocks according to the different data of the different information blocks to obtain an APP application interface that matches the first information.
It can be seen that, according to the embodiment of the application, the data respectively corresponding to the element attributes of the different controls of the other information blocks in the APP interface is obtained by obtaining the data respectively corresponding to the element attributes of the different controls of the first information block in the APP interface; and then adjusting the arrangement of the different information blocks to obtain an APP interface matched with the user requirements. By adopting the means, the data of other information blocks can be acquired according to the data information of the first information block, so that the data crawling method is very convenient, the crawling efficiency is higher, the operation difficulty is lower, and the technical requirement of data crawling is greatly reduced.
The first obtaining module 801 is configured to obtain a presentation layer corresponding to the APP application interface; acquiring data corresponding to a first information block in the display layer and the position of the data of the first information block in the display layer; and acquiring data corresponding to the element attribute of the first control from the data of the first information block, and acquiring the position of the element attribute of the first control in the data of the first information block.
The second obtaining module 802 is configured to obtain data of the second information block according to a position where the data of the first information block is located in the presentation layer; and acquiring data corresponding to the element attribute of the second control from the data of the second information block according to the position of the element attribute of the first control in the data of the first information block.
Further comprises: the confirmation module is used for confirming whether all the information blocks in the APP interface are acquired; and if all the information blocks in the APP interface are acquired, triggering and executing to adjust the arrangement of the different information blocks according to the different data of the different information blocks so as to obtain the APP interface matched with the first information.
The confirmation module is used for confirming whether the display layer corresponding to the APP interface has an end mark or not; if yes, confirming that all information blocks in the APP interface are acquired according to the position of the ending mark.
The confirmation module is further configured to obtain first data in a display layer corresponding to the APP application interface if the display layer corresponding to the APP application interface does not have an end mark; executing a screen sliding program to acquire second data in a display layer corresponding to the APP interface; confirming whether the first data and the second data are consistent; and if the information blocks are consistent, confirming that all the information blocks in the APP interface are acquired.
The data processing module 804 is configured to obtain data related to the first information in different information blocks respectively; sorting the data according to the order from high to low to obtain sorted data; acquiring information blocks respectively corresponding to the sequenced data; and displaying the information blocks respectively corresponding to the sequenced data to an APP interface.
The embodiment of the application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any one of the processing methods of the APP control based on data crawling as described in the embodiment of the method.
Embodiments of the present application also provide a computer program product comprising a non-transitory computer-readable storage medium storing a computer program that causes a computer to perform some or all of the steps of any of the data crawling-based APP control processing methods described in the method embodiments above.
It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present application.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, such as the division of the units, merely a logical function division, and there may be additional manners of dividing the actual implementation, such as multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, or may be in electrical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units described above may be implemented either in hardware or in software program modules.
The integrated units, if implemented in the form of software program modules, may be stored in a computer-readable memory for sale or use as a stand-alone product. Based on this understanding, the technical solution of the present application may be embodied essentially or partly in the form of a software product, or all or part of the technical solution, which is stored in a memory, and includes several instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned memory includes: a U-disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Those of ordinary skill in the art will appreciate that all or a portion of the steps in the various methods of the above embodiments may be implemented by a program that instructs associated hardware, and the program may be stored in a computer readable memory, which may include: flash disk, read-only memory, random access memory, magnetic or optical disk, etc.
The foregoing has outlined rather broadly the more detailed description of embodiments of the application, wherein the principles and embodiments of the application are explained in detail using specific examples, the above examples being provided solely to facilitate the understanding of the method and core concepts of the application; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (8)
1. The processing method of the APP control based on data crawling is characterized by comprising the following steps:
acquiring a display layer corresponding to the APP interface;
acquiring data corresponding to a first information block in the display layer and the position of the data of the first information block in the display layer; the first information block at least comprises a first control;
Acquiring data corresponding to the element attribute of the first control from the data of the first information block, and acquiring the position of the element attribute of the first control in the data of the first information block; the APP interface also includes at least a second information block, the second information block including at least a second control;
acquiring data of the second information block according to the position of the data of the first information block in the display layer;
acquiring data corresponding to the element attribute of the second control from the data of the second information block according to the position of the element attribute of the first control in the data of the first information block;
receiving a data crawling request sent by a terminal, wherein the data crawling request carries first information and is used for indicating to acquire an APP interface matched with the first information;
and adjusting the arrangement of different information blocks according to different data of the different information blocks to obtain an APP interface matched with the first information.
2. The method according to claim 1, wherein after the obtaining the data corresponding to the element attributes of the different controls of the other information blocks in the APP application interface according to the data corresponding to the element attributes of the different controls of the first information block, before the adjusting the arrangement of the different information blocks according to the different data of the different information blocks to obtain the APP application interface matched with the first information, the method comprises:
Confirming whether all information blocks in the APP interface are acquired;
and if all the information blocks in the APP interface are acquired, triggering and executing to adjust the arrangement of the different information blocks according to the different data of the different information blocks so as to obtain the APP interface matched with the first information.
3. The method of claim 2, wherein said confirming whether all information blocks in the APP application interface are acquired comprises:
confirming whether a display layer corresponding to the APP interface has an end mark or not;
if yes, confirming that all information blocks in the APP interface are acquired according to the position of the ending mark.
4. The method of claim 3, wherein if the display layer corresponding to the APP application interface does not have an end mark, the determining whether all the information blocks in the APP application interface are acquired completely comprises:
acquiring first data in a display layer corresponding to the APP interface;
executing a screen sliding program to acquire second data in a display layer corresponding to the APP interface;
confirming whether the first data and the second data are consistent;
And if the information blocks are consistent, confirming that all the information blocks in the APP interface are acquired.
5. The method according to any one of claims 1 to 4, wherein said adjusting the arrangement of the different information blocks according to the different data of the different information blocks to obtain an APP application interface matching the first information comprises:
respectively acquiring data related to the first information in different information blocks;
sorting the data according to the order from high to low to obtain sorted data;
acquiring information blocks respectively corresponding to the sequenced data;
and displaying the information blocks respectively corresponding to the sequenced data to an APP interface.
6. An APP control processing system based on data crawling, comprising:
the first acquisition module is used for acquiring a display layer corresponding to the APP interface, acquiring data of a first information block corresponding to the display layer and the position of the data of the first information block in the display layer; the first information block at least comprises a first control;
the first obtaining module is further configured to obtain data corresponding to an element attribute of the first control from the data of the first information block, and obtain a position of the element attribute of the first control in the data of the first information block; the APP interface also includes at least a second information block, the second information block including at least a second control;
The second acquisition module is used for acquiring the data of the second information block according to the position of the data of the first information block in the display layer, and acquiring the data corresponding to the element attribute of the second control from the data of the second information block according to the position of the element attribute of the first control in the data of the first information block;
the data crawling module is used for receiving a data crawling request sent by a terminal, wherein the data crawling request carries first information and is used for indicating to acquire an APP interface matched with the first information;
and the data processing module is used for adjusting the arrangement of different information blocks according to different data of the different information blocks so as to obtain an APP application interface matched with the first information.
7. A server comprising a processor, an input device, an output device and a memory, the processor, the input device, the output device and the memory being interconnected, wherein the memory is adapted to store a computer program comprising program instructions, the processor being configured to invoke the program instructions to perform the method of any of claims 1 to 5.
8. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program, which is executed by a processor to implement the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910832368.7A CN110569416B (en) | 2019-09-04 | 2019-09-04 | APP control processing method based on data crawling and related products |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910832368.7A CN110569416B (en) | 2019-09-04 | 2019-09-04 | APP control processing method based on data crawling and related products |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110569416A CN110569416A (en) | 2019-12-13 |
| CN110569416B true CN110569416B (en) | 2023-10-13 |
Family
ID=68777746
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910832368.7A Active CN110569416B (en) | 2019-09-04 | 2019-09-04 | APP control processing method based on data crawling and related products |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110569416B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112199568A (en) * | 2020-09-28 | 2021-01-08 | 时趣互动（北京）科技有限公司 | Mobile APP data capturing method, device and system |
| CN112416504A (en) * | 2021-01-04 | 2021-02-26 | 广东创能科技股份有限公司 | Method capable of realizing automatic compatible adaptation for different devices |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102402539A (en) * | 2010-09-15 | 2012-04-04 | 倪毅 | Design technology for object-level personalized vertical search engine |
| WO2015000083A1 (en) * | 2013-07-05 | 2015-01-08 | Anysolution, Inc. | System and method for ranking online content |
| CN108089967A (en) * | 2017-12-12 | 2018-05-29 | 成都睿码科技有限责任公司 | A kind of method for crawling Android mobile phone App data |
| CN110162682A (en) * | 2019-04-12 | 2019-08-23 | 深圳壹账通智能科技有限公司 | A kind of crawling method of network data, device, storage medium and terminal device |
| CN110189189A (en) * | 2019-04-19 | 2019-08-30 | 平安科技（深圳）有限公司 | One-stop shopping at network bootstrap technique, device, computer equipment and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9977790B2 (en) * | 2011-02-04 | 2018-05-22 | Ebay, Inc. | Automatically obtaining real-time, geographically-relevant product information from heterogeneus sources |
| US20180150562A1 (en) * | 2016-11-25 | 2018-05-31 | Cognizant Technology Solutions India Pvt. Ltd. | System and Method for Automatically Extracting and Analyzing Data |
| CN110020068B (en) * | 2017-09-26 | 2021-10-15 | 北京国双科技有限公司 | Method and device for configuring page crawling rules |
| CN109408701B (en) * | 2018-11-08 | 2021-01-15 | 网易（杭州）网络有限公司 | Method and device for displaying crawling path of web crawler |
-
2019
- 2019-09-04 CN CN201910832368.7A patent/CN110569416B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102402539A (en) * | 2010-09-15 | 2012-04-04 | 倪毅 | Design technology for object-level personalized vertical search engine |
| WO2015000083A1 (en) * | 2013-07-05 | 2015-01-08 | Anysolution, Inc. | System and method for ranking online content |
| CN108089967A (en) * | 2017-12-12 | 2018-05-29 | 成都睿码科技有限责任公司 | A kind of method for crawling Android mobile phone App data |
| CN110162682A (en) * | 2019-04-12 | 2019-08-23 | 深圳壹账通智能科技有限公司 | A kind of crawling method of network data, device, storage medium and terminal device |
| CN110189189A (en) * | 2019-04-19 | 2019-08-30 | 平安科技（深圳）有限公司 | One-stop shopping at network bootstrap technique, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110569416A (en) | 2019-12-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN205427915U (en) | Aassessment that is used for of consumer's operation is used for target electronic equipment's of recovery aassessment equipment | |
| CN105897908B (en) | A kind of cloud intelligent cashier system and cash register | |
| CN107251033A (en) | System and method for carrying out active user checking in online education | |
| CN104765715B (en) | cloud font service system | |
| CN102668515B (en) | Relative weighting scheme is applied to use online data | |
| US12014376B2 (en) | Automated verification of user interface process flows | |
| CN110457364B (en) | User information view generation method and device | |
| KR102672533B1 (en) | system and method for automatic investment of financial assets based on quint investment | |
| CN110516173B (en) | Illegal network station identification method, illegal network station identification device, illegal network station identification equipment and illegal network station identification medium | |
| CN110377522A (en) | The test method for scene of trading, calculates equipment and medium at device | |
| US11501322B2 (en) | Blockchain-based data processing systems, methods, and apparatuses | |
| US20140032364A1 (en) | System and method for providing verification of seller authorization and product authentication | |
| CN110569416B (en) | APP control processing method based on data crawling and related products | |
| CN107506662A (en) | Information interacting method and device, computer-readable network storage medium, terminal | |
| CN111008852A (en) | Artwork anti-counterfeiting identification method, server, terminal and storage medium | |
| KR102290481B1 (en) | Method and apparatus for providing real estate transaction service interacting with fintech based on block chain | |
| CN109615471A (en) | Source of houses temperature display methods, device and computer readable storage medium | |
| CN104468635B (en) | The user right step-up authentication method and system of the network platform | |
| CN109493198A (en) | Service evaluation management method, device, system and evaluation server | |
| CN101546343B (en) | Method and device for matching the colors of probes | |
| CN110298751A (en) | Cross-platform transaction data processing method, device, terminal and storage medium | |
| CN114219587B (en) | Order data processing method and device, storage medium and computer equipment | |
| KR20200089485A (en) | Design-technology fusion quality estimating flatform providing method | |
| TWI735866B (en) | Authentication-type agent-based distribution e-commerce system and method thereof | |
| WO2017007149A1 (en) | Automated payment device and operation method therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |