CN108932090A - terminal control method, device and storage medium - Google Patents
terminal control method, device and storage medium Download PDFInfo
- Publication number
- CN108932090A CN108932090A CN201810532088.XA CN201810532088A CN108932090A CN 108932090 A CN108932090 A CN 108932090A CN 201810532088 A CN201810532088 A CN 201810532088A CN 108932090 A CN108932090 A CN 108932090A
- Authority
- CN
- China
- Prior art keywords
- image
- operation object
- control information
- instruction
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04842—Selection of displayed objects or displayed text elements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
- G06V20/42—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items of sport video content
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
This disclosure relates to which a kind of terminal control method, device and storage medium, are related to field of terminal technology, this method includes：By being identified to collected control information, obtain the operation object of control information instruction, and to the operation object object run to be carried out, obtain the first image in current display interface, image recognition is carried out to the first image using preset image recognition algorithm, to determine location information of the operation object in the first image, object run is carried out to operation object on preceding display interface according to location information.It can be improved the feasibility and compatibility of simulated operation, reduce maintenance cost.
Description
Technical field
This disclosure relates to field of terminal technology, and in particular, to a kind of terminal control method, device and storage medium.
Background technique
In the related art, the application with being constantly progressive for terminal technology and being continuously increased for user demand, in terminal
Program (English：Application, referred to as：APP) also increasingly diversified, user can pacify at the terminal according to respective demand
Fill different application programs.However, application program is during realizing user demand, it usually needs operation of the user in terminal
The operation such as clicked, slided or amplified on interface, process is cumbersome, low efficiency, and user experience is bad.It is terminal that simulation, which is clicked,
System obtains the information of page node by number to operation interface or text, thus the operation of analog subscriber, still,
According to the difference of the development tool of application program and platform, such as：OpenGL (English：Open GraphicsLibrary, in
Text：Open graphic library), DirectX (English；Direct extension) etc. platforms, the information of corresponding page node is not yet
Together, and for the application program of third party's exploitation, third party can be adjusted the information of page node at any time.
Summary of the invention
To overcome the problems in correlation technique, the disclosure provides a kind of terminal control method, device and storage medium.
According to the first aspect of the embodiments of the present disclosure, a kind of terminal control method is provided, the method includes：
By identifying to collected control information, the operation object of the control information instruction and right is obtained
The operation object object run to be carried out；
Obtain the first image in current display interface；
Image recognition is carried out to the first image using preset image recognition algorithm, is existed with the determination operation object
Location information in the first image；
The object run is carried out to the operation object on the display interface according to the positional information.
With reference to first aspect, in the first possible implementation, described to utilize preset image recognition algorithm to institute
It states the first image and carries out image recognition, with location information of the determination operation object in the first image, including：
Search in preset object template library with the matched object template of the operation object；
Using the object template and the first image as the input of described image recognizer, according to described image
The output result of recognizer determines the location information of operation object described in the first image；
Wherein, the object template library includes multiple object templates corresponding with a variety of operation objects.
With reference to first aspect the first can realize mode, described to utilize preset figure in second of achievable mode
Picture recognizer carries out image recognition to the first image, with position of the determination operation object in the first image
Information further includes：
Search in preset order button template library with the matched order button template of the object run；
Using described instruction button template and the first image as the input of described image recognizer, according to
The output result of image recognition algorithm determines the position of the corresponding order button of object run described in the first image；
Wherein, described instruction button template library includes multiple described instruction button moulds corresponding with a variety of object runs
Plate.
The achievable mode of second with reference to first aspect, the third can be described according to the position in realization mode
Information carries out the object run to the operation object on the display interface, including：
According to the position of described instruction button and the location information, to the operation object on the display interface into
The row object run.
With reference to first aspect, in the 4th kind of achievable mode, it is described using preset image recognition algorithm to described the
One image carries out image recognition, with location information of the determination operation object in the first image, including：
The first image is identified using described image recognizer, to determine in the first image whether deposit
In the operation object；
When there are when the operation object, obtaining the position of operation object described in the first image in the first image
Confidence breath；
When the operation object is not present in the first image, the display reminding information on the display interface, institute
It states prompt information and is used to indicate operation failure.
With reference to first aspect any one of to the 4th kind of possible implementation of first aspect, can be achieved at the 5th kind
It is described by being identified to collected control information in mode, the operation object of the acquisition control information instruction, and
To the operation object object run to be carried out, including：
When the control information is voice messaging, by carrying out speech recognition to the collected control information, obtain
The operation object for taking the control information to indicate, and to the operation object object run to be carried out；
When the control information is action message, by carrying out action recognition to the collected control information, obtain
The operation object for taking the control information to indicate, and to the operation object object run to be carried out.
According to the second aspect of an embodiment of the present disclosure, a kind of terminal control mechanism is provided, described device includes：
Data obtaining module is configured as obtaining the control information by identifying collected control information
The operation object of instruction, and to the operation object object run to be carried out；
Image collection module is configured as obtaining the first image in current display interface；
Identification module is configured as carrying out image recognition to the first image using preset image recognition algorithm, with
Determine location information of the operation object in the first image；
Operation module is configured as carrying out institute to the operation object on the display interface according to the positional information
State object run.
In conjunction with second aspect, in the first possible implementation, the identification module includes：
First searches submodule, is configured as searching in preset object template library and the matched object of the operation object
Template；
It identifies submodule, is configured as using the object template and the first image as described image recognizer
Input determines the position letter of operation object described in the first image with the output result according to described image recognizer
Breath；
Wherein, the object template library includes multiple object templates corresponding with a variety of operation objects.
The first in conjunction with second aspect can realize mode, and in second of achievable mode, the identification module is also wrapped
It includes：
Second searches submodule, is configured as searching matched with the object run in preset order button template library
Order button template；
The identification submodule, is additionally configured to using described instruction button template and the first image as described image
The input of recognizer determines object run described in the first image with the output result according to described image recognizer
The position of corresponding order button；
Wherein, described instruction button template library includes multiple described instruction button moulds corresponding with a variety of object runs
Plate.
, can be in realization mode at the third in conjunction with second of achievable mode of second aspect, the operation module is matched
It is set to：
According to the position of described instruction button and the location information, to the operation object on the display interface into
The row object run.
In conjunction with second aspect, in the 4th kind of achievable mode, the identification module includes：
It determines submodule, is configured as identifying the first image using described image recognizer, with determination
It whether there is the operation object in the first image；
Acquisition submodule is configured as when there are when the operation object, obtaining first figure in the first image
The location information of the operation object as described in；
Prompting submodule is configured as when the operation object is not present in the first image, in display circle
Display reminding information on face, the prompt information are used to indicate operation failure.
In conjunction with any one of the 4th kind of possible implementation of second aspect to second aspect, can be achieved at the 5th kind
In mode, the data obtaining module includes：
First acquisition submodule is configured as when the control information is voice messaging, by collected described
It controls information and carries out speech recognition, obtain the operation object of the control information instruction, and to the operation object institute
The object run to be carried out；
Second acquisition submodule is configured as when the control information is action message, by collected described
It controls information and carries out action recognition, obtain the operation object of the control information instruction, and to the operation object institute
The object run to be carried out.
According to the third aspect of an embodiment of the present disclosure, a kind of terminal control mechanism is provided, including：
Processor；
Memory for storage processor executable instruction；
Wherein, the processor is configured to running the instruction in the memory to realize that disclosure first aspect is mentioned
The step of terminal control method of confession.
According to a fourth aspect of embodiments of the present disclosure, a kind of computer readable storage medium is provided, calculating is stored thereon with
Machine program instruction, which is characterized in that realize that disclosure first aspect is mentioned when the computer program instructions are executed by processor
The step of terminal control method of confession
Through the above technical solutions, the disclosure passes through the collected control information of identification first, to obtain control information
In indicated operation object, and object run corresponding with operation object, then obtain first currently shown on display interface
Image determines location information of the operation object in the first image, last basis by carrying out image recognition to the first image
Location information of the operation object in the first image carries out object run to operation object, makes the user do not need to click manually, so that it may
To realize the operation of simulation click, and it is not only restricted to the application program of third party's exploitation, can be improved the feasibility of simulated operation
And compatibility, reduce maintenance cost.
Other feature and advantage of the disclosure will the following detailed description will be given in the detailed implementation section.
Detailed description of the invention
Attached drawing is and to constitute part of specification for providing further understanding of the disclosure, with following tool
Body embodiment is used to explain the disclosure together, but does not constitute the limitation to the disclosure.In the accompanying drawings：
Fig. 1 is a kind of flow chart of terminal control method shown according to an exemplary embodiment；
Fig. 2 is the flow chart of another terminal control method shown according to an exemplary embodiment；
Fig. 3 is the flow chart of another terminal control method shown according to an exemplary embodiment；
Fig. 4 is the flow chart of another terminal control method shown according to an exemplary embodiment；
Fig. 5 is the flow chart of another terminal control method shown according to an exemplary embodiment；
Fig. 6 is a kind of block diagram of terminal control mechanism shown according to an exemplary embodiment；
Fig. 7 is the block diagram of another terminal control mechanism shown according to an exemplary embodiment；
Fig. 8 is the block diagram of another terminal control mechanism shown according to an exemplary embodiment；
Fig. 9 is the block diagram of another terminal control mechanism shown according to an exemplary embodiment；
Figure 10 is the block diagram of another terminal control mechanism shown according to an exemplary embodiment；
Figure 11 is a kind of block diagram of terminal control mechanism shown according to an exemplary embodiment.
Specific embodiment
Example embodiments are described in detail here, and the example is illustrated in the accompanying drawings.Following description is related to
When attached drawing, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements.Following exemplary embodiment
Described in embodiment do not represent all implementations consistent with this disclosure.On the contrary, they be only with it is such as appended
The example of the consistent device and method of some aspects be described in detail in claims, the disclosure.
Before introducing the terminal control method, device and storage medium of disclosure offer, first to each in the disclosure
Application scenarios involved in embodiment are introduced, which includes a terminal, are provided in the terminal for display figure
Picture, the display interface for receiving user operation instruction.The terminal can be set to Android (English：Android) system, iOS system,
The operating system of the types such as Windows Phone system, Mac OS system, Windows system, linux system, the disclosure is to end
The operating system at end is without limitation.The terminal for example can be smart phone, tablet computer, smart television, smartwatch, PDA
(Personal Digital Assistant, personal digital assistant), portable computer, VR (English：Virtual Reality,
Chinese：Virtual reality) glasses, AR (English：Augmented Reality, Chinese：Augmented reality) mobile terminals such as glasses,
It can be the fixed terminals such as desktop computer.
Fig. 1 is a kind of flow chart of terminal control method shown according to an exemplary embodiment, as shown in Figure 1, the party
Method includes：
Step 101, it by being identified to collected control information, obtains and controls the operation object that information indicates, with
And to the operation object object run to be carried out.
It for example, can be by information collecting device come real-time monitoring user hair during user's using terminal
Control information out, wherein information collecting device may include for acquire the voice acquisition device of user speech (such as：Words
Cylinder) and for acquiring user action (gesture, eyeball mobile etc.) image collecting device (such as：Camera) at least one
Kind, corresponding, control information may include：User passes through voice, gesture, at least one on the move of eyeball.According to difference
The control information of type carries out speech recognition or action recognition, obtains the operation object and corresponding target indicated in control information
Operation.Identification to control information, can first pass through in advance a large amount of sample collection to determine, can also be according to the specific need of user
It asks to be arranged.Such as：The voice of user can be largely acquired, generating sound bank can also be by user to identify control information
The mobile trend of fixed gesture or eyeball is set and corresponds to different demands.Wherein, operation object is to be used to indicate user to need
The title for the specific object to be operated, object run are which type of behaviour for being used to indicate user and needing to carry out operation object
Make, can be specific operation (such as：Select, click, double-click, amplify, reduce, drag), it is also possible to operate some
Button operation (such as：Confirming button, cancel button).It is illustrated with user in use " fighting landlord " application program, user thinks
The card of a heart 5 out has issued the voice (i.e. control information) of " heart 5 ", identifies user's language by speech recognition technology
Sound analyzes the semanteme for including in user speech, and obtaining middle operation object is：" heart 5 ", object run in the state of missing,
It can be defaulted as " playing a card ".
It is arranged at the terminal it should be noted that the information collecting device of acquisition control information can be, is also possible to
It can be communicated with terminal, independent electronic equipment, it, can be by mutual after information collecting device collects control information
Networking, WLAN (English：Wireless Local Area Networks, Chinese：WLAN), such as Wi-Fi (English：
Wireless-Fidelity, Chinese：Wireless Fidelity), bluetooth (English：Bluetooth) or BLE is (English：Bluetooth Low
Energy, Chinese：Bluetooth Low Energy) etc. modes by control information be sent to terminal.
Step 102, the first image in current display interface is obtained.
It is exemplary, according to the type of the operating system of terminal, corresponding screenshotss function or screenshotss script are called, to display circle
Current image carries out screenshotss operation on face, obtains the first image.By taking Android system as an example, call
ScreenshotUtils.getScreenshot function obtains the bitmap file (English in current display interface：Bitmap).
Step 103, the first image is identified using preset image recognition algorithm, to determine operation object first
Location information in image.
Step 104, object run is carried out to operation object in the display interface according to location information.
For example, using the first image as the input of preset image recognition algorithm, the output of image recognition algorithm can
To include number (identify several operation objects) and location information of the operation object in the first image, wherein location information can
To include：The coordinate of operation object on the first image.Image recognition algorithm can be according to the hardware computing capability of terminal, and knows
The different demands of other precision select template matching algorithm, SSD (English：Single Shot MultiBox Detector, in
Text：The more box detectors of single-shot) algorithm, YOLO (English：You Only Look Once) algorithm or Faster-RCNN (English：
Faster-Regions with Convolutional Neural Network features) image recognition algorithms such as algorithm.
After determining location information, by calling corresponding function in terminal operating system, according to object run in the first image
Operation object, which is operated, (such as can be：Simulation clicks, simulation dragging, Simulation scale-up etc.), it, can be with by taking Android system as an example
Realize that simulation is clicked using Instrumentation function.
It is illustrated with user in use " mah-jong pieces " application program, user finds out the mah-jong pieces of one " east wind ", have issued
The voice of " beating east wind ", obtaining operation object by speech recognition technology is " east wind ", and object run is " playing a card ", active user
There are the mah-jong pieces of three " east wind " in hand, then including three target objects (" east in the first image in current display interface
Wind "), the position of three target objects in the first image is identified by image recognition algorithm, any " east wind " is clicked in simulation
It plays a card, to realize the purpose of simulation clicking operation under the premise of being not required to user's progress click manually.
In conclusion the disclosure passes through the collected control information of identification first, to obtain indicated in control information
Operation object, and object run corresponding with operation object, then obtain the first image currently shown on display interface passes through
Image recognition is carried out to determine location information of the operation object in the first image to the first image, is finally existed according to operation object
Location information in first image carries out object run to operation object, makes the user do not need to click manually, so that it may realize simulation
The operation of click, and it is not only restricted to the application program of third party's exploitation, it can be improved the feasibility and compatibility of simulated operation, drop
Low-maintenance cost.
Fig. 2 is the flow chart of another terminal control method shown according to an exemplary embodiment, as shown in Fig. 2, step
Rapid 103 include：
Step 1031, search in preset object template library with the matched object template of operation object.
Step 1032, using object template and the first image as the input of image recognition algorithm, to be calculated according to image recognition
The output result of method determines the location information of operation object in the first image.
Wherein, object template library includes multiple object templates corresponding with a variety of operation objects.
For example, it is searched and the matched object template of operation object first in preset object template library, wherein right
As template is the image information of operation object, such as the manipulable operation object of whole in application program can be mentioned in advance
It takes out, as object template library, object template library can also be determined by a large amount of sample collection, with " fighting landlord " application
Program is illustrated, and may include the corresponding image information of 54 cards, i.e. 54 object templates in object template library.It can also be into
54 object templates are divided by one step according to different patterns：Plum blossom template, square template, heart template and spade template, or
54 object templates are divided into according to different digital：1 template, 2 template ..., the template of K.Again by object template and first
Input of the image as image recognition algorithm, to identify the position of operation object in the first image.
In the above-described embodiment, by by preset object template library with the matched object template of operation object and
Input of one image as image recognition algorithm can accurately be known to obtain the location information of operation object in the first image
Not Chu operation object position, to improve the accuracy of simulated operation.
Fig. 3 is the flow chart of another terminal control method shown according to an exemplary embodiment, as shown in figure 3, step
Rapid 103 further include：
Step 1033, search in preset order button template library with the matched order button template of object run.
Step 1034, using order button template and the first image as the input of image recognition algorithm, to be known according to image
The output result of other algorithm determines the position of the corresponding order button of object run in the first image.
Wherein, order button template library includes multiple instruction button template corresponding with plurality of target operation.
It is exemplary, it is identical as embodiment shown in Fig. 2, it is searched first in preset order button template library and target is grasped
Make matched order button template, wherein order button template is the image information of order button, such as will can be applied in advance
The manipulable instruction of whole in program extracts, and as order button template library, is illustrated with " 21 points " application program,
May include in order button template library：The order buttons such as suspension, hit, double.Again by order button template and the first image
As the input of image recognition algorithm, to identify the position of the corresponding order button of object run in the first image.
In the above-described embodiment, by by preset order button template library with the matched order button of object run
The input of template and the first image as image recognition algorithm, to obtain the corresponding order button of object run in the first image
Position, the position of order button can be recognized accurately, to further increase the accuracy of simulated operation.
Optionally, step 104 can be accomplished by the following way：
According to the position of order button and location information, object run is carried out to the operation object on display interface.
It is exemplary, it is illustrated with user in use " mah-jong pieces " application program, user finds out the mah-jong pieces of one " 30,000 ",
The voice for having issued " making a call to 30,000 ", obtaining operation object by speech recognition technology is " 30,000 ", and object run is " playing a card ", from
Through obtaining the position of the location information and " playing a card " corresponding button of playing a card of " 30,000 " in the first image in first image, then mould
It is quasi- to click " 30,000 ", then button of playing a card is clicked, to realize that simulation is clicked.
It should be noted that user, which issues in control information, may indicate that multiple operation objects, with user in use " bucket
Landlord " application program is illustrated, and user finds out two 7 cards, have issued " out to 7 " (can also be " to 7 ", " two 7 " or
" a pair 7 " etc.) voice, user speech is identified by speech recognition technology, the semanteme for including in user speech is analyzed, is grasped
It is as object：The card of " 7 ", the number of operation object are two, and object run is " playing a card ".So obtaining the first image
Afterwards, 7 template (including four kinds of patterns) is found in the object template library of 54 cards, then is determined by image recognition algorithm
The position of location information corresponding to 7 card and the order button of object run corresponding (button of playing a card) in first image, if
There are three 7 cards in active user's hand, then the location information of available three operation objects, finally simulation is clicked any
Two 7 cards, then simulate click and play a card button, get two 7.
In the above-described embodiment, it by the position of order button and location information, is selected in the display interface to realize
Determine operation object, and carry out object run, can accurately realize the demand of user, to improve the feasibility of simulated operation.
Fig. 4 is the flow chart of another terminal control method shown according to an exemplary embodiment, as shown in figure 4, step
Rapid 103 include：
Step 1035, the first image is identified using image recognition algorithm, is whether there is with determining in the first image
Operation object.
Step 1036, when there are when operation object, obtaining the location information of operation object in the first image in the first image.
Step 1037, when operation object is not present in the first image, display reminding information, prompts letter in the display interface
Breath is used to indicate operation failure.
For example, after the operation object identified, it is possible that operation object can not be matched with the first image
The scene of operation object is not present on i.e. the first image for problem, and terminal cannot achieve virtual click at this time, therefore, can first benefit
It is determined with image recognition algorithm with the presence or absence of operation object in the first image, if it does, obtaining operation pair in the first image again
As corresponding location information, if it does not, can display reminding information (can be with the loudspeaking of controlling terminal in the display interface
Device issues prompt tone), to prompt user, mobile the issued instruction operation failure of this voice, gesture or eyeball.
In the above-described embodiment, first by the first image of identification, to determine that operation object whether there is, when operation pair
As in the presence of, simulated operation is normally executed, in the absence of operation object, sends prompt information to inform user, operation failure,
It can identify that can control information execute, select corresponding processing mode for different scenes, to improve the strong of simulated operation
Strong property.
Fig. 5 is the flow chart of another terminal control method shown according to an exemplary embodiment, as shown in figure 5, step
Rapid 101 include：
Step 1011, it when controlling information is voice messaging, by carrying out speech recognition to collected control information, obtains
The operation object of control information instruction is taken, and to the operation object object run to be carried out.
Step 1012, it when controlling information is action message, by carrying out action recognition to collected control information, obtains
The operation object of control information instruction is taken, and to the operation object object run to be carried out.
Exemplary, control information can be divided into two class of voice messaging and action message, corresponding voice acquisition device and image
Acquisition device can carry out voice knowledge to control information according to pre-set sound bank when controlling information is voice messaging
Not, when controlling information is action message (gesture, eyeball movement etc.), control can be believed according to pre-set maneuver library
Breath carries out action recognition.It is illustrated with user using player viewing video, " waving " can be arranged in user according to demand
Movement is corresponding " playing video ", movement of " clenching fist " corresponding " pause video " etc., when image collecting device captures " waving "
When, determine that operation object is the video in player according to action recognition, object run is " broadcasting ".
In the above-described embodiment, for different types of control information, identifying processing is carried out respectively, when control information is
When voice messaging, speech recognition is carried out, when controlling information is action message, action recognition is carried out, different type can be directed to
Control information select corresponding processing mode, to improve the robustness of simulated operation.
Fig. 6 is a kind of block diagram of terminal control mechanism shown according to an exemplary embodiment, as shown in fig. 6, the device
200 include：
Data obtaining module 201 is configured as obtaining control information by identifying collected control information and referring to
The operation object shown, and to the operation object object run to be carried out.
Image collection module 202 is configured as obtaining the first image in current display interface.
Identification module 203 is configured as carrying out image recognition to the first image using preset image recognition algorithm, with true
Determine location information of the operation object in the first image.
Operation module 204 is configured as carrying out object run to operation object in the display interface according to location information.
Fig. 7 is the block diagram of another terminal control mechanism shown according to an exemplary embodiment, as shown in fig. 7, identification
Module 203 includes：
First searches submodule 2031, is configured as searching in preset object template library and the matched object of operation object
Template.
It identifies submodule 2032, is configured as using object template and the first image as the input of image recognition algorithm, with
The location information of operation object described in the first image is determined according to the output result of image recognition algorithm.
Wherein, object template library includes multiple object templates corresponding with a variety of operation objects.
Fig. 8 is the block diagram of another terminal control mechanism shown according to an exemplary embodiment, as shown in figure 8, identification
Module 203 further includes：
Second searches submodule 2033, is configured as searching matched with object run in preset order button template library
Order button template.
It identifies submodule 2032, is additionally configured to using order button template and the first image as the defeated of image recognition algorithm
Enter, the position of the corresponding order button of object run in the first image is determined with the output result according to image recognition algorithm.
Wherein, order button template library includes multiple instruction button template corresponding with plurality of target operation.
Optionally, operation module 204 can be configured as：
According to the position of order button and location information, object run is carried out to the operation object on display interface.
Fig. 9 is the block diagram of another terminal control mechanism shown according to an exemplary embodiment, as shown in figure 9, identification
Module 203 includes：
It determines submodule 2034, is configured as identifying the first image using image recognition algorithm, to determine first
It whether there is operation object in image.
Acquisition submodule 2035 is configured as when there are when operation object, obtaining and operate in the first image in the first image
The location information of object.
Prompting submodule 2036 is configured as showing in the display interface when operation object is not present in the first image
Prompt information, prompt information are used to indicate operation failure.
Figure 10 is the block diagram of another terminal control mechanism shown according to an exemplary embodiment, as shown in Figure 10, letter
Breath obtains module 201：
First acquisition submodule 2011 is configured as when controlling information is voice messaging, by collected control
Information carries out speech recognition, obtains the operation object of control information instruction, and to the operation object object run to be carried out.
Second acquisition submodule 2012 is configured as when controlling information is action message, by collected control
Information carries out action recognition, obtains the operation object of control information instruction, and to the operation object object run to be carried out.
About the device in above-described embodiment, wherein modules execute the concrete mode of operation in related this method
Embodiment in be described in detail, no detailed explanation will be given here.
In conclusion the disclosure passes through the collected control information of identification first, to obtain indicated in control information
Operation object, and object run corresponding with operation object, then obtain the first image currently shown on display interface passes through
Image recognition is carried out to determine location information of the operation object in the first image to the first image, is finally existed according to operation object
Location information in first image carries out object run to operation object, makes the user do not need to click manually, so that it may realize simulation
The operation of click, and it is not only restricted to the application program of third party's exploitation, it can be improved the feasibility and compatibility of simulated operation, drop
Low-maintenance cost.
Figure 11 is a kind of block diagram of terminal control mechanism 300 shown according to an exemplary embodiment.For example, device 300
It can be mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, Medical Devices,
Body-building equipment, personal digital assistant etc..
Referring to Fig.1 1, device 300 may include following one or more components：Processing component 302, memory 304, electric power
Component 306, multimedia component 308, audio component 310, the interface 312 of input/output (I/O), sensor module 314, and
Communication component 316.
The integrated operation of the usual control device 300 of processing component 302, such as with display, telephone call, data communication, phase
Machine operation and record operate associated operation.Processing component 302 may include that one or more processors 320 refer to execute
It enables, to complete all or part of the steps of above-mentioned terminal control method.In addition, processing component 302 may include one or more
A module, convenient for the interaction between processing component 302 and other assemblies.For example, processing component 302 may include multimedia mould
Block, to facilitate the interaction between multimedia component 308 and processing component 302.
Memory 304 is configured as storing various types of data to support the operation in device 300.These data are shown
Example includes the instruction of any application or method for operating on the device 300, contact data, and telephone book data disappears
Breath, picture, video etc..Memory 304 can be by any kind of volatibility or non-volatile memory device or their group
It closes and realizes, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM) is erasable to compile
Journey read-only memory (EPROM), programmable read only memory (PROM), read-only memory (ROM), magnetic memory, flash
Device, disk or CD.
Electric power assembly 306 provides electric power for the various assemblies of device 300.Electric power assembly 306 may include power management system
System, one or more power supplys and other with for device 300 generate, manage, and distribute the associated component of electric power.
Multimedia component 308 includes the screen of one output interface of offer between described device 300 and user.One
In a little embodiments, screen may include liquid crystal display (LCD) and touch panel (TP).If screen includes touch panel, screen
Curtain may be implemented as touch screen, to receive input signal from the user.Touch panel includes one or more touch sensings
Device is to sense the gesture on touch, slide, and touch panel.The touch sensor can not only sense touch or sliding action
Boundary, but also detect duration and pressure associated with the touch or slide operation.In some embodiments, more matchmakers
Body component 308 includes a front camera and/or rear camera.When device 300 is in operation mode, such as screening-mode or
When video mode, front camera and/or rear camera can receive external multi-medium data.Each front camera and
Rear camera can be a fixed optical lens system or have focusing and optical zoom capabilities.
Audio component 310 is configured as output and/or input audio signal.For example, audio component 310 includes a Mike
Wind (MIC), when device 300 is in operation mode, when such as call mode, recording mode, and voice recognition mode, microphone is matched
It is set to reception external audio signal.The received audio signal can be further stored in memory 304 or via communication set
Part 316 is sent.In some embodiments, audio component 310 further includes a loudspeaker, is used for output audio signal.
I/O interface 312 provides interface between processing component 302 and peripheral interface module, and above-mentioned peripheral interface module can
To be keyboard, click wheel, button etc..These buttons may include, but are not limited to：Home button, volume button, start button and lock
Determine button.
Sensor module 314 includes one or more sensors, and the state for providing various aspects for device 300 is commented
Estimate.For example, sensor module 314 can detecte the state that opens/closes of device 300, and the relative positioning of component, for example, it is described
Component is the display and keypad of device 300, and sensor module 314 can be with 300 1 components of detection device 300 or device
Position change, the existence or non-existence that user contacts with device 300,300 orientation of device or acceleration/deceleration and device 300
Temperature change.Sensor module 314 may include proximity sensor, be configured to detect without any physical contact
Presence of nearby objects.Sensor module 314 can also include optical sensor, such as CMOS or ccd image sensor, at
As being used in application.In some embodiments, which can also include acceleration transducer, gyro sensors
Device, Magnetic Sensor, pressure sensor or temperature sensor.
Communication component 316 is configured to facilitate the communication of wired or wireless way between device 300 and other equipment.Device
300 can access the wireless network based on communication standard, such as WiFi, 2G or 3G or their combination.In an exemplary implementation
In example, communication component 316 receives broadcast singal or broadcast related information from external broadcasting management system via broadcast channel.
In one exemplary embodiment, the communication component 316 further includes near-field communication (NFC) module, to promote short range communication.Example
Such as, NFC module can be based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra wide band (UWB) technology,
Bluetooth (BT) technology and other technologies are realized.
In the exemplary embodiment, device 300 can be believed by one or more application specific integrated circuit (ASIC), number
Number processor (DSP), digital signal processing appts (DSPD), programmable logic device (PLD), field programmable gate array
(FPGA), controller, microcontroller, microprocessor or other electronic components are realized, for executing above-mentioned terminal control method.
In the exemplary embodiment, a kind of non-transitorycomputer readable storage medium including instruction, example are additionally provided
It such as include the memory 304 of instruction, above-metioned instruction can be executed by the processor 320 of device 300 to complete above-mentioned terminal control
Method.For example, the non-transitorycomputer readable storage medium can be ROM, random access memory (RAM), CD-ROM,
Tape, floppy disk and optical data storage devices etc..
In conclusion the disclosure passes through the collected control information of identification first, to obtain indicated in control information
Operation object, and object run corresponding with operation object, then obtain the first image currently shown on display interface passes through
Image recognition is carried out to determine location information of the operation object in the first image to the first image, is finally existed according to operation object
Location information in first image carries out object run to operation object, makes the user do not need to click manually, so that it may realize simulation
The operation of click, and it is not only restricted to the application program of third party's exploitation, it can be improved the feasibility and compatibility of simulated operation, drop
Low-maintenance cost.
The preferred embodiment of the disclosure is described in detail in conjunction with attached drawing above, still, the disclosure is not limited to above-mentioned reality
The detail in mode is applied, in the range of the technology design of the disclosure, a variety of letters can be carried out to the technical solution of the disclosure
Monotropic type, these simple variants belong to the protection scope of the disclosure.
It is further to note that specific technical features described in the above specific embodiments, in not lance
In the case where shield, it can be combined in any appropriate way.In order to avoid unnecessary repetition, the disclosure to it is various can
No further explanation will be given for the combination of energy.
In addition, any combination can also be carried out between a variety of different embodiments of the disclosure, as long as it is without prejudice to originally
Disclosed thought equally should be considered as disclosure disclosure of that.
Claims (14)
1. a kind of terminal control method, which is characterized in that the method includes：
By identifying to collected control information, the operation object of the control information instruction is obtained, and to described
The operation object object run to be carried out；
Obtain the first image in current display interface；
Image recognition is carried out to the first image using preset image recognition algorithm, with the determination operation object described
Location information in first image；
The object run is carried out to the operation object on the display interface according to the positional information.
2. the method according to claim 1, wherein described utilize preset image recognition algorithm to described first
Image carries out image recognition, with location information of the determination operation object in the first image, including：
Search in preset object template library with the matched object template of the operation object；
Using the object template and the first image as the input of described image recognizer, to be identified according to described image
The output result of algorithm determines the location information of operation object described in the first image；
Wherein, the object template library includes multiple object templates corresponding with a variety of operation objects.
3. according to the method described in claim 2, it is characterized in that, described utilize preset image recognition algorithm to described first
Image carries out image recognition with location information of the determination operation object in the first image：
Search in preset order button template library with the matched order button template of the object run；
Using described instruction button template and the first image as the input of described image recognizer, according to described image
The output result of recognizer determines the position of the corresponding order button of object run described in the first image；
Wherein, described instruction button template library includes multiple described instruction button templates corresponding with a variety of object runs.
4. according to the method described in claim 3, it is characterized in that, described according to the positional information on the display interface
The object run is carried out to the operation object, including：
According to the position of described instruction button and the location information, institute is carried out to the operation object on the display interface
State object run.
5. the method according to claim 1, wherein described utilize preset image recognition algorithm to described first
Image carries out image recognition, with location information of the determination operation object in the first image, including：
The first image is identified using described image recognizer, to determine in the first image with the presence or absence of institute
State operation object；
When there are the position letters for when operation object, obtaining operation object described in the first image in the first image
Breath；
When the operation object is not present in the first image, display reminding information, described to mention on the display interface
Show that information is used to indicate operation failure.
6. method according to any one of claims 1-5, which is characterized in that described by collected control information
It is identified, obtains the operation object of the control information instruction, and to the operation object object run to be carried out,
Including：
When the control information is voice messaging, by carrying out speech recognition to the collected control information, institute is obtained
The operation object of control information instruction is stated, and to the operation object object run to be carried out；
When the control information is action message, by carrying out action recognition to the collected control information, institute is obtained
The operation object of control information instruction is stated, and to the operation object object run to be carried out.
7. a kind of terminal control mechanism, which is characterized in that described device includes：
Data obtaining module is configured as obtaining the control information instruction by identifying collected control information
Operation object, and to the operation object object run to be carried out；
Image collection module is configured as obtaining the first image in current display interface；
Identification module is configured as carrying out image recognition to the first image using preset image recognition algorithm, with determination
Location information of the operation object in the first image；
Operation module is configured as carrying out the mesh to the operation object on the display interface according to the positional information
Mark operation.
8. device according to claim 7, which is characterized in that the identification module includes：
First searches submodule, is configured as searching in preset object template library and the matched object mould of the operation object
Plate；
It identifies submodule, is configured as using the object template and the first image as the defeated of described image recognizer
Enter, the location information of operation object described in the first image is determined with the output result according to described image recognizer；
Wherein, the object template library includes multiple object templates corresponding with a variety of operation objects.
9. device according to claim 8, which is characterized in that the identification module further includes：
Second searches submodule, is configured as searching in preset order button template library and the matched instruction of the object run
Button template；
The identification submodule is additionally configured to identify using described instruction button template and the first image as described image
The input of algorithm determines that object run described in the first image is corresponding with the output result according to described image recognizer
Order button position；
Wherein, described instruction button template library includes multiple described instruction button templates corresponding with a variety of object runs.
10. device according to claim 9, which is characterized in that the operation module is configured as：
According to the position of described instruction button and the location information, institute is carried out to the operation object on the display interface
State object run.
11. device according to claim 7, which is characterized in that the identification module includes：
It determines submodule, is configured as identifying the first image using described image recognizer, described in determination
It whether there is the operation object in first image；
Acquisition submodule is configured as when there are when the operation object, obtaining in the first image in the first image
The location information of the operation object；
Prompting submodule is configured as when the operation object is not present in the first image, on the display interface
Display reminding information, the prompt information are used to indicate operation failure.
12. device according to any one of claims 7-11, which is characterized in that the data obtaining module includes：
First acquisition submodule is configured as when the control information is voice messaging, by the collected control
Information carries out speech recognition, obtains the operation object of the control information instruction, and will be into the operation object
Capable object run；
Second acquisition submodule is configured as when the control information is action message, by the collected control
Information carries out action recognition, obtains the operation object of the control information instruction, and will be into the operation object
Capable object run.
13. a kind of terminal control mechanism, which is characterized in that including：
Processor；
Memory for storage processor executable instruction；
Wherein, the processor is configured to running the instruction in the memory to realize any one of claim 1-6 institute
The step of stating method.
14. a kind of computer readable storage medium, is stored thereon with computer program instructions, which is characterized in that the computer
The step of any one of claim 1-6 the method is realized when program instruction is executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810532088.XA CN108932090A (en) | 2018-05-29 | 2018-05-29 | terminal control method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810532088.XA CN108932090A (en) | 2018-05-29 | 2018-05-29 | terminal control method, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108932090A true CN108932090A (en) | 2018-12-04 |
Family
ID=64449446
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810532088.XA Pending CN108932090A (en) | 2018-05-29 | 2018-05-29 | terminal control method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108932090A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109745699A (en) * | 2018-12-29 | 2019-05-14 | 维沃移动通信有限公司 | A kind of method and terminal device responding touch control operation |
| CN110738185A (en) * | 2019-10-23 | 2020-01-31 | 腾讯科技（深圳）有限公司 | Form object identification method and device and storage medium |
| CN111013138A (en) * | 2019-12-11 | 2020-04-17 | 苏州思必驰信息科技有限公司 | Voice control method and device, electronic equipment and storage medium |
| CN113741769A (en) * | 2020-05-29 | 2021-12-03 | 比亚迪股份有限公司 | Control method and system based on image recognition, vehicle and storage medium |
| CN116756045A (en) * | 2023-08-14 | 2023-09-15 | 海马云（天津）信息技术有限公司 | Application testing method and device, computer equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104899003A (en) * | 2015-06-12 | 2015-09-09 | 广州视源电子科技股份有限公司 | Terminal control method and system |
| CN104965596A (en) * | 2015-07-24 | 2015-10-07 | 上海宝宏软件有限公司 | Voice control system |
| US20160018909A1 (en) * | 2014-07-16 | 2016-01-21 | Alibaba Group Holding Limited | Method and apparatus of controlling a smart device |
| CN106201177A (en) * | 2016-06-24 | 2016-12-07 | 维沃移动通信有限公司 | A kind of operation execution method and mobile terminal |
| CN106843669A (en) * | 2016-12-06 | 2017-06-13 | 北京小度信息科技有限公司 | Application interface operating method and device |
| CN107024988A (en) * | 2017-03-20 | 2017-08-08 | 宇龙计算机通信科技(深圳)有限公司 | A kind of method and device that operation is performed based on user action |
-
2018
- 2018-05-29 CN CN201810532088.XA patent/CN108932090A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160018909A1 (en) * | 2014-07-16 | 2016-01-21 | Alibaba Group Holding Limited | Method and apparatus of controlling a smart device |
| CN104899003A (en) * | 2015-06-12 | 2015-09-09 | 广州视源电子科技股份有限公司 | Terminal control method and system |
| CN104965596A (en) * | 2015-07-24 | 2015-10-07 | 上海宝宏软件有限公司 | Voice control system |
| CN106201177A (en) * | 2016-06-24 | 2016-12-07 | 维沃移动通信有限公司 | A kind of operation execution method and mobile terminal |
| CN106843669A (en) * | 2016-12-06 | 2017-06-13 | 北京小度信息科技有限公司 | Application interface operating method and device |
| CN107024988A (en) * | 2017-03-20 | 2017-08-08 | 宇龙计算机通信科技(深圳)有限公司 | A kind of method and device that operation is performed based on user action |
Non-Patent Citations (1)
| Title |
|---|
| 车云: "《智能汽车：决战20202》", 30 April 2018 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109745699A (en) * | 2018-12-29 | 2019-05-14 | 维沃移动通信有限公司 | A kind of method and terminal device responding touch control operation |
| CN110738185A (en) * | 2019-10-23 | 2020-01-31 | 腾讯科技（深圳）有限公司 | Form object identification method and device and storage medium |
| CN110738185B (en) * | 2019-10-23 | 2023-07-07 | 腾讯科技（深圳）有限公司 | Form object identification method, form object identification device and storage medium |
| CN111013138A (en) * | 2019-12-11 | 2020-04-17 | 苏州思必驰信息科技有限公司 | Voice control method and device, electronic equipment and storage medium |
| CN113741769A (en) * | 2020-05-29 | 2021-12-03 | 比亚迪股份有限公司 | Control method and system based on image recognition, vehicle and storage medium |
| CN116756045A (en) * | 2023-08-14 | 2023-09-15 | 海马云（天津）信息技术有限公司 | Application testing method and device, computer equipment and storage medium |
| CN116756045B (en) * | 2023-08-14 | 2023-10-31 | 海马云（天津）信息技术有限公司 | Application testing method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108932090A (en) | terminal control method, device and storage medium | |
| US20220044056A1 (en) | Method and apparatus for detecting keypoints of human body, electronic device and storage medium | |
| CN109117862B (en) | Image tag recognition methods, device and server | |
| CN104852966B (en) | Numeric value transfer, terminal and cloud server | |
| CN104850828B (en) | Character recognition method and device | |
| CN111368796B (en) | Face image processing method and device, electronic equipment and storage medium | |
| WO2021136975A1 (en) | Image processing methods and apparatuses, electronic devices, and storage media | |
| CN110674719A (en) | Target object matching method and device, electronic equipment and storage medium | |
| CN109359056A (en) | A kind of applied program testing method and device | |
| CN110231901A (en) | Application interface display methods and device | |
| CN107766820A (en) | Image classification method and device | |
| EP3291225A1 (en) | Method and device for securely adding connection | |
| CN109672830A (en) | Image processing method, device, electronic equipment and storage medium | |
| US20210201478A1 (en) | Image processing methods, electronic devices, and storage media | |
| CN104662889B (en) | Method and apparatus for being shot in portable terminal | |
| CN109410276A (en) | Key point position determines method, apparatus and electronic equipment | |
| CN107832746A (en) | Expression recognition method and device | |
| CN109543536A (en) | Image identification method and device, electronic equipment and storage medium | |
| CN114514566B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN110532956A (en) | Image processing method and device, electronic equipment and storage medium | |
| CN109685041A (en) | Image analysis method and device, electronic equipment and storage medium | |
| CN109657539A (en) | Face value evaluation method and device, readable storage medium and electronic equipment | |
| CN109388699A (en) | Input method, device, equipment and storage medium | |
| CN107560611A (en) | Moving line acquisition methods and device | |
| CN112437231B (en) | Image shooting method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181204 |
|
| RJ01 | Rejection of invention patent application after publication |