CN110032503A - Data processing system, method, equipment and device based on UI automation and OCR - Google Patents
Data processing system, method, equipment and device based on UI automation and OCR Download PDFInfo
- Publication number
- CN110032503A CN110032503A CN201811307223.7A CN201811307223A CN110032503A CN 110032503 A CN110032503 A CN 110032503A CN 201811307223 A CN201811307223 A CN 201811307223A CN 110032503 A CN110032503 A CN 110032503A
- Authority
- CN
- China
- Prior art keywords
- application program
- data information
- interest
- data processing
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 32
- 238000000034 method Methods 0.000 title claims abstract description 24
- 239000000284 extract Substances 0.000 claims abstract description 22
- 238000000605 extraction Methods 0.000 claims description 10
- 238000003672 processing method Methods 0.000 claims description 8
- 238000012937 correction Methods 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 description 6
- 238000012360 testing method Methods 0.000 description 6
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 235000019013 Viburnum opulus Nutrition 0.000 description 4
- 244000071378 Viburnum opulus Species 0.000 description 4
- 230000008901 benefit Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 101100055496 Arabidopsis thaliana APP2 gene Proteins 0.000 description 2
- 102100039424 Polyadenylate-binding protein 4 Human genes 0.000 description 2
- 101100016250 Saccharomyces cerevisiae (strain ATCC 204508 / S288c) GYL1 gene Proteins 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 241000406668 Loxodonta cyclotis Species 0.000 description 1
- BUGBHKTXTAQXES-UHFFFAOYSA-N Selenium Chemical compound [Se] BUGBHKTXTAQXES-UHFFFAOYSA-N 0.000 description 1
- 230000009194 climbing Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000009193 crawling Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 201000006549 dyspepsia Diseases 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 229910052711 selenium Inorganic materials 0.000 description 1
- 239000011669 selenium Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Multimedia (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The present invention relates to data processing system, method, equipment and devices based on UI automation and OCR, the system (10) includes: terminal (100), it is configured as regularly respectively enteing one or more user interfaces about object of interest by means of UI automatic operation one or more application program, and obtains one or more image corresponding with one or more of user interfaces；And processor (120,220), it is configured as that OCR recognizer is called to identify respectively and extract data information relevant to the object of interest in the one or more image.Using the system, method, equipment and device, it can be no longer limited by the available resources of API in data grabber and call frequency and number.
Description
Technical field
The present invention relates to Internet technical fields, more particularly to based on UI (user interface) automation and OCR (optical character
Identification) data processing system, method, equipment and device.
Background technique
Web crawlers be it is a kind of according to certain rules, automatically grab the program or script of web message.Network
The major way of crawler is to carry out data grabber by the API (application programming interface) that website provides.The shortcomings that passing through API
It is the number and frequency that some websites will limit API Calls, user needs to pay more flexible to obtain to upgrade into advanced level user
API Calls.
The another way of web crawlers is that data grabber is carried out based on HTML, and advantage is exactly not limited by the calling of API
System.By accessing the HTML code of webpage, and therefrom grab the data on required node.There are one disadvantages for this method just
It is that once the structure change of little bit occurs, crawl code is possible to need to rewrite webpage.
Summary of the invention
In view of the problem above of the prior art, the embodiment of the present invention provides a kind of novel based on UI automation and OCR
Data processing system, method, equipment and device, be no longer limited by the available resources of API and call frequency and number.
For market business, multi-source comparison is to guarantee the continual and steady correct available effective means of market data.With
The development of terminal technology and software technology, current market data multi-source alignment schemes need to analyze other market data
The interface rules of data source on APP (application program), crawl his source data by interface.But it after interface change, needs pair
Logic follow-up modification is crawled, to cause maintenance cost height.In addition, the promotion of the anti-ability of crawling with other systems, can divide
The data source that interface is precipitated is also more and more limited.
On the other hand, OCR (optical character identification) refers to that electronic equipment determines it by mode dark, bright in detection image
Then shape is translated into the process of computword by shape with character identifying method.But OCR technique often relies on use
Family is for the scanning of content to be identified or specifying for images to be recognized.Thus, OCR technique at present can't be automatically to big spirogram
As carrying out Text region.
Automatic test refers to that the automation of software test, software test are exactly operating system or application on a preset condition based
Program assesses operation result, and preparatory condition should include normal condition and exceptional condition.Automatic test includes that unit automation is surveyed
Try (data analysis layer), interface automatic test (Business Logic) and UI automatic test (gui interface layer).
It is all to be presented to use on market data APP that present inventor, which notices the market data to be crawled usually,
Shown data in the user interface at family.In turn, present inventor is proposed first by means of UI automatization simulation user
It operates market data APP and enters user interface to be captured, screenshotss call OCR identification service, identify the text envelope in screenshotss
Breath, then extracts the content for needing to check field, can finally be compared to more market data of crawl.Especially,
Truncated picture is aggregated into cloud, by the powerful data counts ability of server, data grabber can be greatlyd improve
Efficiency.Term " UI automation " refers to by writing program, realizes to the interactive actions such as the electric shock of element, dragging on APP, thus
Realize the process for utilizing machine simulation user's operation APP.In this way, the data grabber of embodiment according to the present invention is no longer limited by
The available resources and calling frequency and number of API.
A kind of data processing system based on UI automation and OCR of embodiment according to the invention, comprising: it is configured
Regularly to be respectively enterd one or more about object of interest by means of UI automatic operation one or more application program
A user interface, and obtain one or more image corresponding with one or more of user interfaces；And processor,
It is configured as that OCR recognizer is called to identify and extract related to the object of interest in the one or more image respectively
Data information.
A kind of data processing method based on UI automation and OCR of embodiment according to the invention, comprising: regularly borrow
Help the one or more application program in UI automatic operation terminal and respectively enters one or more about object of interest
User interface；Obtain one or more image corresponding with one or more of user interfaces；Also, OCR identification is called to calculate
Method identifies respectively and extracts data information relevant to the object of interest in the one or more image.
A kind of data processing equipment based on UI automation and OCR of embodiment according to the invention, comprising: processor；
And memory, it is stored thereon with executable instruction, wherein the executable instruction makes the processor upon being performed
Execute method above-mentioned.
A kind of machine readable storage medium of embodiment according to the invention, is stored thereon with executable instruction, wherein institute
State executable instruction makes machine execute method above-mentioned upon being performed.
A kind of data processing equipment based on UI automation and OCR of embodiment according to the invention, comprising: obtain mould
Block, for regularly being respectively enterd by means of the one or more application program in UI automatic operation terminal about interested right
One or more user interfaces of elephant, and obtain one or more image corresponding with one or more of user interfaces；With
And extraction module, for call OCR recognizer to identify respectively and extract in the one or more image with it is described interested
The relevant data information of object.
From the above, it can be seen that the scheme of the embodiment of the present invention enters specified use using UI automatic operation application program
Family interface, screenshotss can obtain related data, crawl without analyzing his source interface, it is not easy to be prohibited by the anti-system of climbing of other side
Only.Therefore, compared with prior art, what the scheme of the embodiment of the present invention can be no longer limited by API when data crawl can
With resource and call frequency and number.After frame according to an embodiment of the present invention is built simultaneously, it can be adapted for stock, base
The comparison of golden various businesses data, without by the variation of various businesses data-interface greatly and the biggish shadow of data structure mapping
It rings, significantly reduces maintenance cost.In addition, also solve in the prior art can only be from service for the scheme of the embodiment of the present invention
Whether also end checks whether market data correctly can be used, be unable to ensure client and show correct problem.In addition, according to the present invention
Technology can be applied not only to the scene of market comparing, can be also used for intelligent monitoring analyze competing product APP activity and
State.
Detailed description of the invention
Feature of the invention, feature, advantage and benefit will become obvious by the detailed description below in conjunction with attached drawing.
Fig. 1 shows the data processing system 10 based on UI automation and OCR of one embodiment according to the invention
Exemplary scene.
Fig. 2 shows the data processing methods 300 based on UI automation and OCR of one embodiment according to the invention
Overview flow chart.
Fig. 3 shows the data processing equipment 400 based on UI automation and OCR of one embodiment according to the invention
Schematic diagram.
Fig. 4 shows Alipay (ant wealth), rich way ox ox, five sections of snowball, straight flush and self-selected stock market data APP
Synchronization is obtained about the long-living stock information of ST.
Specific embodiment
Theme described herein is discussed referring now to example embodiment.It should be understood that discussing these embodiments only
It is in order to enable those skilled in the art can better understand that being not to claim to realize theme described herein
Protection scope, applicability or the exemplary limitation illustrated in book.It can be in the protection scope for not departing from present disclosure
In the case of, the function and arrangement of the element discussed are changed.Each example can according to need, omit, substitute or
Add various processes or component.For example, described method can be executed according to described order in a different order, with
And each step can be added, omits or combine.In addition, feature described in relatively some examples is in other examples
It can be combined.
As used in this article, term " includes " and its modification indicate open term, are meant that " including but not limited to ".
Term "based" indicates " being based at least partially on ".Term " one embodiment " and " embodiment " expression " at least one implementation
Example ".Term " another embodiment " expression " at least one other embodiment ".Term " first ", " second " etc. may refer to not
Same or identical object.Here may include other definition, either specific or implicit.Unless bright in context
It really indicates, otherwise the definition of a term is consistent throughout the specification.
In the following, in conjunction with each embodiment of attached drawing detailed description of the present invention.
Fig. 1 shows the data processing system 10 based on UI automation and OCR of one embodiment according to the invention.Figure
System 10 shown in 1 includes terminal 100, has been equipped with one or more application program APP1, APP2 in advance on the terminal 100
Deng.Terminal 100 can be the client mobile communication end of mobile phone, tablet computer etc., be also possible to that the application can be run
The desktop computer or laptop of program.For stock market scene, the application program may include, but be not limited to pay
The market data APP such as treasured, self-selected stock, snowball, straight flush, rich way ox ox.It is described to apply journey for intelligent customer service robot scene
Sequence may include, but be not limited to the intelligent customer services robots such as Alipay intelligent customer service robot, Jingdone district intelligent customer service robot
APP.Those skilled in the art will appreciate that after reading and understanding present specification other application scenarios of the invention with
And the application program under various application scenarios.
In exemplary scene shown in Fig. 1, terminal 100 is configured as regularly (for example, each day of trade, each transaction
Per every other hour or a few houres during period, trading session, etc.) by means of UI automatic operation one or more application program
APP1, APP2 ... respectively enter one or more user interfaces about object of interest, and obtain with it is one or more of
The corresponding one or more image of user interface.For stock market scene, the object of interest can be user's care
Certain stock or stock code.The common testing tool of UI automatic test have UFT, Robot Framework, Selenium,
Appium etc.." the UI automation " can be programmed for specific market data APP.For example, if described interested
Object is set to " Alibaba ", and after terminal 100 automates login " self-selected stock " APP by means of UI, automatically clicking is most upper
Side search column and input " Alibaba ", obtain stock search result card (a kind of graphical user circle in relation to " Alibaba "
Face).Meanwhile terminal 100 can automate by means of UI and log in " Alipay " APP, after " wealth " of clicking bottom, click most
The search column of top simultaneously automatically enters " Alibaba ", finally obtains the stock search result card in relation to " Alibaba ".It can be with
After the results page of search successfully renders, the corresponding image of search result card is obtained by taking pictures.
In exemplary scene shown in Fig. 1, terminal 100 includes one or more processors 120, is configured as calling
OCR recognizer identifies respectively and extracts data information relevant to the object of interest in the one or more image.
Known OCR recognizer is all available.For example, processor 120 can be identified and be mentioned by available OCR recognizer
It takes in the search result image for acquisition of taking pictures and for example as " Alibaba " of object of interest related market data.
In exemplary scene shown in Fig. 1, terminal 100 further includes memory 110.If search result card by screenshotss,
Memory 110 can store the screenshotss.Thus, screenshotss image in the accessible memory 110 of processor 120 and by available
OCR recognizer the screenshotss image stored is identified and is extracted.
In exemplary scene shown in Fig. 1, data processing system 10 further includes cloud server 200, independently of terminal
100, and including memory 210 and one or more processors 220.Terminal 100 will be obtained by way of wirelessly or non-wirelessly
The one or more image taken sends the memory 210 of server 200 to, and memory 210 stores the image received.Example
Such as, terminal 100 can by screenshotss content uploading to Operation Support System (OSS) and obtain image URL (unified resource position
Symbol), then the URL of image is sent to the memory 210 of server 200.In this case, by the processing of server 200
Device 220 realizes the identification and extraction to described image.For example, processor 220 judges whether this takes turns the time interval of movement images
Meet pre-provisioning request；If meeting pre-provisioning request, available OCR recognizer is just called, converts text information for image information,
Extract the data (for example, current share prices of " Alibaba ") of corresponding field.Thus it is possible to make full use of the powerful of server
Operational capability fast and efficiently identifies and extracts data information relevant to the object of interest in described image.
Under the scene for carrying out multi-source comparison to market data, processor 120,220 be can be configured as to from multiple
Application program (such as Alipay, self-selected stock, snowball, straight flush, rich way ox ox etc.) with the object of interest (such as ST long
It is raw) relevant data information (such as current share prices) is compared.Fig. 4 shows Alipay (ant wealth), rich way ox ox, snow
Five sections of ball, straight flush and self-selected stock market data APP synchronizations are obtained about the long-living stock information of ST.In such case
Under, processor 120,220, which can be configured as, calls available OCR recognizer, comes from Alipay (ant from shown in Fig. 4
Wealth), rich way ox ox, snowball, straight flush and self-selected stock five width images in identify and extract the long-living current share prices of ST, go forward side by side
Row laterally compares, to judge whether various market data APP are identical for the share price of same B shareB.If processor 120,
220 determine that a certain or certain several market data APP (are greater than one with other for such as displaying share price of designated speculative stock
It is a) the displaying share price of market data APP is inconsistent, then signal an alert.For example, if alarm signal indicates Alipay (ant
Ant wealth) long-living about the ST long-living share price of four ST of stock price and other is inconsistent, then exploitation tester can be with
Start the inspection for Alipay APP, where finding the problem.
In addition, although the case where OCR recognition capability is continuously improved, and there are still wrong identifications at present.According to this hair
Bright embodiment, if processor 120,220 passes through to the image for identical object of interest from a plurality of application programs
Recognition result, which is compared, determines that the recognition result greater than a application program is consistent, then processor 120,220 can make
The image recognition result of those inconsistent application programs is corrected with the image recognition result from these application programs.This
Sample can improve the identification quality of OCR recognizer used.
In addition, technology according to the present invention can be applied not only to the scene of market comparing, intelligence can be also used for
The activity and state of the competing product APP of monitoring analysis.For example, terminal 100 can be configured as by the way of UI automation, open all
It such as the application program of Jingdone district finance APP, clicks to enter that bank is selected in homepage, shows one in the selected page of bank of expansion
A little Determined products and its earning rate.In addition, the marketing position of rolling also can use UI automation and be clicked, into page
Screenshot uploads behind the inside of face.Processor 120,220 can call available OCR recognizer to identify sense from the image obtained
The information of interest.Then, processor 120,220 can by recognition result that this is obtained from a application program and it is last from
The recognition result obtained with a application program is compared.If consistent, just without alarm；If inconsistent (for example, competing product
The new product of APP restocking or operation marketing activity), processor 120,220 can send alarm at this time, to notify related personnel
It goes to and carries out checking analysis.
Those skilled in the art will appreciate that, the above-mentioned institute that processor is realized it is functional can by individual processor Lai
Implement, can also be implemented respectively by multiple processors.
Fig. 2 shows the data processing methods 300 based on UI automation and OCR of one embodiment according to the invention
Overview flow chart.The data processing method 300 of Fig. 2 is described in detail below with reference to data processing system 10 shown in FIG. 1.
As shown in Fig. 2, in box 310, regularly by means of the one or more application in UI automatic operation terminal 100
Program respectively enters one or more user interfaces about object of interest.
In box 320, one or more image corresponding with one or more of user interfaces is obtained.
In box 340, if obtaining the one or more image by way of taking pictures, for example, by processor 120,
220, call OCR recognizer to identify respectively and extract number relevant to the object of interest in the one or more image
It is believed that breath.
In box 330, if obtaining the one or more image, acquired one or more by way of screenshotss
Image can be stored locally in the memory 110 of terminal 100, or can be stored remotely server 200 beyond the clouds
In memory 210.Then, in box 340, processor 120,220 calls the OCR recognizer to identify respectively and extracts and stored
Data information relevant to the object of interest in one or more image.
In box 350, for example, by processor 120,220, to from the multiple application program with it is described interested
The relevant data information of object is compared.In a further embodiment, the method can also include by this extract with
The relevant data information of the object of interest and it is last extract from same application domain with the object of interest
Relevant data information is compared.
It is corrected based on the result of the comparison from the multiple in box 360 for example, by processor 120,220
The data information of one or more application program in application program.
In box 370, for example, by processor 120,220, the multiple application is come from the result instruction of the comparison
The data information of one or more application program is different from from other application journey in the multiple application program in program
When the data information of sequence, alarm signal is exported.In a further embodiment, the method can also be included in the comparison
Result when indicating the data information that this is extracted and the inconsistent last data information that extracts, export about institute
State the alarm signal of application program.
It can be seen from the above that the scheme of the present embodiment is using market data APP objectively to market data
Displaying is realized based on UI automation and OCR identification technology and can be carried out multi-source data comparison not through api interface.Meanwhile
Also it can improve the identification quality of existing OCR.Although of the invention above with respect to the embodiment of the present invention of market data description
Design also can be applied to obtain any scenes of other data sources.For example, for intelligent customer service robot scene, it is desirable to comment
The problem of system, can be inputed to such as Alipay intelligent customer service robot and competing by Gu Jingpin intelligent customer service robot response rate
Product intelligent customer service robot, after getting the answer of robot, screenshotss are uploaded, analyses and comparison Alipay intelligent customer service robot
With the answer of Jing Pin intelligent customer service robot, and the knowledge base of Alipay intelligent customer service robot can be expanded, promote product
Competitiveness and user experience.
Other modifications
It will be appreciated by those skilled in the art that although in the above embodiments, data processing system 10 includes terminal
100, however, the present invention is not limited thereto.In some other embodiment of the invention, the number based on UI automation and OCR
It can only include memory 110,210 and processor 120,220 according to processing equipment.It is stored on memory 110,210 executable
Instruction, wherein the executable instruction makes processor 120,220 execute method 300 shown in Fig. 2 upon being performed.It is described
Data processing equipment can be independently of terminal 100.Moreover, processor 120,220, which can be configured as controlling terminal 100, is based on UI
The one or more of application programs of automatic operation.In some embodiments, memory 110,210 be additionally configured to storage by
The one or more image that terminal 100 transmits by way of wirelessly or non-wirelessly, for identifying and extracting.
Fig. 3 shows the data processing equipment 400 based on UI automation and OCR of one embodiment according to the invention
Schematic diagram.Device 400 shown in Fig. 3 can use the mode of software, hardware or software and hardware combining to realize.
As shown in figure 3, device 400 may include obtaining module 410 and extraction module 430.Module 410 is obtained for regular
Ground respectively enterd by means of the one or more application program in UI automatic operation terminal about one of object of interest or
Multiple user interfaces, and obtain one or more image corresponding with one or more of user interfaces.Extraction module 430 is used
It is identified respectively in calling OCR recognizer and extracts data relevant to the object of interest in the one or more image
Information.
In a first aspect, device 400 further includes memory module 420, for an acquired width to be stored locally or remotely
Or multiple image.In this case, the extraction module is also used to that OCR recognizer is called to identify and extract respectively and is stored
One or more image in data information relevant to the object of interest.
In second aspect, device 400 further includes comparison module 440, for from the multiple application program with institute
The relevant data information of object of interest is stated to be compared.In a further embodiment, comparison module 440 can be also used for this
The data information relevant to the object of interest of secondary extraction with last time extract from same application domain with it is described
The relevant data information of object of interest is compared.
In the third aspect, device 400 further includes correction module 450, for being come from based on the result of the comparison to correct
The data information of one or more application program in the multiple application program.
In fourth aspect, device 400 further includes alarm module 460, for the result instruction in the comparison from described
The data information of one or more application program is different from coming from its in the multiple application program in multiple application programs
When the data information of his application program, alarm signal is exported.In a further embodiment, alarm module 460 can also be used
Indicate that the data information that this is extracted and the upper data information once extracted are inconsistent in the result in the comparison
When, export the alarm signal about the application program.
Embodiment according to the invention also provides a kind of machine readable storage medium, is stored thereon with executable instruction,
In, the executable instruction makes machine execute method 300 shown in Fig. 2 upon being performed.
The specific embodiment illustrated above in conjunction with attached drawing describes exemplary embodiment, it is not intended that may be implemented
Or fall into all embodiments of the protection scope of claims." exemplary " meaning of the term used in entire this specification
Taste " be used as example, example or illustration ", be not meant to than other embodiments " preferably " or " there is advantage ".For offer pair
The purpose of the understanding of described technology, specific embodiment include detail.However, it is possible in these no details
In the case of implement these technologies.In some instances, public in order to avoid the concept to described embodiment causes indigestion
The construction and device known is shown in block diagram form.
The foregoing description of present disclosure is provided so that any those of ordinary skill in this field can be realized or make
Use present disclosure.To those skilled in the art, the various modifications carried out to present disclosure are apparent
, also, can also answer generic principles defined herein in the case where not departing from the protection scope of present disclosure
For other modifications.Therefore, present disclosure is not limited to examples described herein and design, but disclosed herein with meeting
Principle and novel features widest scope it is consistent.
Claims (16)
1. a kind of data processing system (10) based on UI automation and OCR, comprising:
Terminal (100) is configured as regularly respectively enteing pass by means of UI automatic operation one or more application program
In one or more user interfaces of object of interest, and obtain a width or more corresponding with one or more of user interfaces
Width image；And
Processor (120,220) is configured as that OCR recognizer is called to identify respectively and extract the one or more image
In data information relevant to the object of interest.
2. data processing system (10) as described in claim 1,
Wherein, the processor (220) is located in cloud server (200), and
Wherein, the data processing system (10) further includes the memory (210) at the cloud server (200), institute
It states memory and is configured as the one or more image that storage is transmitted by way of wirelessly or non-wirelessly by the terminal (100),
For identifying and extracting.
3. data processing system (10) as described in claim 1, further includes:
Memory (110) is configured as that acquired one or more image is locally stored, for identifying and extracting, preferably
Wherein, the memory (110) and the processor (120) are all located in the terminal (100) on ground.
4. the data processing system (10) as described in any one of claim 1-3,
Wherein, the processor (120,220) is additionally configured to execute at least one of following movement:
Data information relevant with the object of interest from the multiple application program is compared,
The number of the one or more application program in the multiple application program is corrected based on the result of the comparison
It is believed that breath, and
The data letter of the one or more application program in the multiple application program is indicated in the result of the comparison
Breath is different from exporting alarm signal in the multiple application program when the data information of other applications.
5. the data processing system (10) as described in any one of claim 1-3,
Wherein, the processor (120,220) is additionally configured to execute at least one of following movement:
What the data information relevant to the object of interest and last time that this is extracted extracted comes from same application domain
Data information relevant with the object of interest be compared, and
Indicate that the data information that this is extracted and the upper data information once extracted are different in the result of the comparison
When cause, the alarm signal about the application program is exported.
6. a kind of data processing method (300) based on UI automation and OCR, comprising:
Regularly respectively enterd by means of the one or more application program in UI automatic operation terminal about object of interest
One or more user interfaces (310)；
Obtain one or more image (320) corresponding with one or more of user interfaces；And
OCR recognizer is called to identify respectively and extract number relevant to the object of interest in the one or more image
It is believed that breath (340).
7. data processing method (300) as claimed in claim 6, further includes:
Acquired one or more image (330) is stored locally or remotely,
Wherein, described to identify and extract including calling OCR recognizer to identify respectively and extracting stored one or more figure
The data information relevant to the object of interest as in.
8. data processing method (300) as claimed in claims 6 or 7, further includes at least one following:
(350) are compared to the data information relevant with the object of interest from the multiple application program,
The number of the one or more application program in the multiple application program is corrected based on the result of the comparison
It is believed that breath (360), and
The data letter of the one or more application program in the multiple application program is indicated in the result of the comparison
Breath is different from exporting alarm signal (370) in the multiple application program when the data information of other applications.
9. data processing method (300) as claimed in claims 6 or 7, further includes at least one following:
What the data information relevant to the object of interest and last time that this is extracted extracted comes from same application domain
Data information relevant with the object of interest be compared, and
Indicate that the data information that this is extracted and the upper data information once extracted are different in the result of the comparison
When cause, the alarm signal about the application program is exported.
10. a kind of data processing equipment based on UI automation and OCR, comprising:
Processor (120,220)；And
Memory (110,210), is stored thereon with executable instruction, wherein the executable instruction makes institute upon being performed
It states processor perform claim and requires method described in any one of 6-9.
11. data processing equipment as claimed in claim 10,
Wherein, the terminal is independently of the data processing equipment,
It wherein, is by the data processing equipment to the operation of one or more of application programs based on UI automation
It is that the processor remotely executes or by the processor execution in the terminal, and
Wherein, the memory is additionally configured to store the width transmitted by way of wirelessly or non-wirelessly by the terminal or more
Width image, for identifying and extracting.
12. a kind of data processing equipment (400) based on UI automation and OCR, comprising:
It obtains module (410), for regularly being distinguished by means of the one or more application program in UI automatic operation terminal
Into one or more user interfaces about object of interest, and obtain corresponding with one or more of user interfaces one
Width or multiple image；And
Extraction module (430), for call OCR recognizer to identify respectively and extract in the one or more image with it is described
The relevant data information of object of interest.
13. data processing equipment (400) as claimed in claim 12, further includes:
Memory module (420), for acquired one or more image to be stored locally or remotely,
Wherein, the extraction module is also used to that OCR recognizer is called to identify respectively and extracts stored one or more figure
The data information relevant to the object of interest as in.
14. data processing equipment (400) as described in claim 12 or 13, further includes at least one following:
Comparison module (440), for the data information relevant to the object of interest from the multiple application program
It is compared,
Correction module (450), for correcting in the multiple application program one or more based on the result of the comparison
The data information of a application program, and
Alarm module (460), for being answered in the instruction of the result of the comparison from one or more in the multiple application program
It is different from the multiple application program when the data information of other applications with the data information of program,
Export alarm signal.
15. data processing equipment (400) as described in claim 12 or 13, further includes at least one following:
Comparison module (440), data information relevant to the object of interest and last extraction for extracting this
The data information relevant with the object of interest from same application domain be compared, and
Alarm module (460), for indicating the data information and upper primary extraction that this is extracted in the result of the comparison
The data information it is inconsistent when, export alarm signal about the application program.
16. a kind of machine readable storage medium, is stored thereon with executable instruction, wherein the executable instruction, which is worked as, to be performed
When make machine perform claim require method described in any one of 6-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811307223.7A CN110032503A (en) | 2018-11-05 | 2018-11-05 | Data processing system, method, equipment and device based on UI automation and OCR |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811307223.7A CN110032503A (en) | 2018-11-05 | 2018-11-05 | Data processing system, method, equipment and device based on UI automation and OCR |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110032503A true CN110032503A (en) | 2019-07-19 |
Family
ID=67234979
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811307223.7A Pending CN110032503A (en) | 2018-11-05 | 2018-11-05 | Data processing system, method, equipment and device based on UI automation and OCR |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110032503A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112256636A (en) * | 2020-11-10 | 2021-01-22 | 国网湖南省电力有限公司 | Data acquisition system for mobile application APP |
| CN112329900A (en) * | 2020-11-05 | 2021-02-05 | 惠科股份有限公司 | Control method and device for cleaning photoresist pipeline and computer readable storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102214349A (en) * | 2011-06-13 | 2011-10-12 | 广东工业大学 | Unattended stock staring and ordering system and ordering method thereof |
| CN102915437A (en) * | 2011-08-02 | 2013-02-06 | 腾讯科技（深圳）有限公司 | Text information identification method and system |
| CN105095207A (en) * | 2014-04-18 | 2015-11-25 | 阿里巴巴集团控股有限公司 | Methods for retrieving and obtaining contents of application software, and devices for retrieving and obtaining contents of application software |
| CN105988924A (en) * | 2015-02-10 | 2016-10-05 | 中国船舶工业综合技术经济研究院 | Automatic testing method for non-intrusive type embedded software graphical user interface |
| CN106293365A (en) * | 2015-05-20 | 2017-01-04 | 广州爱九游信息技术有限公司 | A kind of method and device obtaining content of pages |
| CN106354482A (en) * | 2015-07-13 | 2017-01-25 | 西安中兴新软件有限责任公司 | Method and device for acquiring automated scripts and application method and device of automated scripts |
| CN108536597A (en) * | 2018-04-11 | 2018-09-14 | 上海达梦数据库有限公司 | A kind of terminal test method, device, terminal device and storage medium |
| US10078679B1 (en) * | 2010-09-27 | 2018-09-18 | Trulia, Llc | Verifying the validity and status of data from disparate sources |
| US10095925B1 (en) * | 2017-12-18 | 2018-10-09 | Capital One Services, Llc | Recognizing text in image data |
-
2018
- 2018-11-05 CN CN201811307223.7A patent/CN110032503A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10078679B1 (en) * | 2010-09-27 | 2018-09-18 | Trulia, Llc | Verifying the validity and status of data from disparate sources |
| CN102214349A (en) * | 2011-06-13 | 2011-10-12 | 广东工业大学 | Unattended stock staring and ordering system and ordering method thereof |
| CN102915437A (en) * | 2011-08-02 | 2013-02-06 | 腾讯科技（深圳）有限公司 | Text information identification method and system |
| CN105095207A (en) * | 2014-04-18 | 2015-11-25 | 阿里巴巴集团控股有限公司 | Methods for retrieving and obtaining contents of application software, and devices for retrieving and obtaining contents of application software |
| CN105988924A (en) * | 2015-02-10 | 2016-10-05 | 中国船舶工业综合技术经济研究院 | Automatic testing method for non-intrusive type embedded software graphical user interface |
| CN106293365A (en) * | 2015-05-20 | 2017-01-04 | 广州爱九游信息技术有限公司 | A kind of method and device obtaining content of pages |
| CN106354482A (en) * | 2015-07-13 | 2017-01-25 | 西安中兴新软件有限责任公司 | Method and device for acquiring automated scripts and application method and device of automated scripts |
| US10095925B1 (en) * | 2017-12-18 | 2018-10-09 | Capital One Services, Llc | Recognizing text in image data |
| CN108536597A (en) * | 2018-04-11 | 2018-09-14 | 上海达梦数据库有限公司 | A kind of terminal test method, device, terminal device and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 罗锋: "基于图像识别技术的电子衡器检定数据自动化处理***的设计", 《衡器》 * |
| 黄彦等: "基于多数据源校验的电网EMS数据质量在线监控方法及其实现", 《电力***保护与控制》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112329900A (en) * | 2020-11-05 | 2021-02-05 | 惠科股份有限公司 | Control method and device for cleaning photoresist pipeline and computer readable storage medium |
| CN112256636A (en) * | 2020-11-10 | 2021-01-22 | 国网湖南省电力有限公司 | Data acquisition system for mobile application APP |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA3018196C (en) | Visual regresssion testing tool | |
| US10620975B1 (en) | GUI element acquisition using a plurality of alternative representations of the GUI element | |
| TW201737163A (en) | Problem prediction method and prediction system | |
| US11042472B2 (en) | Authoring automated test suites using artificial intelligence | |
| US20150227276A1 (en) | Method and system for providing an interactive user guide on a webpage | |
| US20200225927A1 (en) | Methods and systems for automating computer application tasks using application guides, markups and computer vision | |
| CN112182359A (en) | Feature management method and system of recommendation model | |
| CN105868225A (en) | Data query method, client, server and system | |
| US20180260099A1 (en) | Smart Display Data Capturing Platform For Record Systems | |
| TW202121206A (en) | Method and system for automatically identifying valid data acquisition module | |
| CN110032503A (en) | Data processing system, method, equipment and device based on UI automation and OCR | |
| CN109062799A (en) | Regression testing method, the apparatus and system of advertising scenarios | |
| CN116720489B (en) | Page filling method and device, electronic equipment and computer readable storage medium | |
| CN107368407A (en) | Information processing method and device | |
| US20130191895A1 (en) | Recovery of Information from Commercial Web Portals | |
| CN110276183B (en) | Reverse Turing verification method and device, storage medium and electronic equipment | |
| US20190087503A1 (en) | Indexable and searchable replay of website user behavior | |
| US9032281B2 (en) | System and method for collecting financial information over a global communications network | |
| US20220237108A1 (en) | Source traceability-based impact analysis | |
| CN115904527A (en) | Data processing method, device, equipment and medium | |
| CN115130041A (en) | Webpage quality evaluation method, neural network training method, device and equipment | |
| US11507907B2 (en) | Multi-computer processing system with machine learning engine for optimized forecasting | |
| US10699105B2 (en) | Workflow creation by image analysis | |
| EP3852032A1 (en) | Machine first approach for identifying accessibility, non-compliances, remediation techniques and fixing at run-time | |
| CN114115628A (en) | U shield display information acquisition method, device, equipment, medium and program product applied to U shield test |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20201012 Address after: English genus Applicant after: Innovative advanced technology Co.,Ltd. Address before: English genus Applicant before: Advanced innovation technology Co.,Ltd. Effective date of registration: 20201012 Address after: English genus Applicant after: Advanced innovation technology Co.,Ltd. Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands Applicant before: Alibaba Group Holding Ltd. |
|
| TA01 | Transfer of patent application right | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190719 |
|
| RJ01 | Rejection of invention patent application after publication |