US20170289582A1 - Device control method and electric device - Google Patents
Device control method and electric device Download PDFInfo
- Publication number
- US20170289582A1 US20170289582A1 US15/622,126 US201715622126A US2017289582A1 US 20170289582 A1 US20170289582 A1 US 20170289582A1 US 201715622126 A US201715622126 A US 201715622126A US 2017289582 A1 US2017289582 A1 US 2017289582A1
- Authority
- US
- United States
- Prior art keywords
- input device
- speech
- user
- recommended
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims abstract description 195
- 230000007613 environmental effect Effects 0.000 claims abstract description 4
- 230000008569 process Effects 0.000 description 105
- 238000010586 diagram Methods 0.000 description 42
- 238000004891 communication Methods 0.000 description 16
- 238000004590 computer program Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 10
- 230000008901 benefit Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 7
- 230000008859 change Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 102100034761 Cilia- and flagella-associated protein 418 Human genes 0.000 description 1
- 101100439214 Homo sapiens CFAP418 gene Proteins 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L17/00—Speaker identification or verification techniques
- G10L17/22—Interactive procedures; Man-machine interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
-
- G—PHYSICS
- G08—SIGNALLING
- G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
- G08C17/00—Arrangements for transmitting signals characterised by the use of a wireless electrical link
-
- G—PHYSICS
- G08—SIGNALLING
- G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
- G08C17/00—Arrangements for transmitting signals characterised by the use of a wireless electrical link
- G08C17/02—Arrangements for transmitting signals characterised by the use of a wireless electrical link using a radio link
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/233—Processing of audio elementary streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/258—Client or end-user data management, e.g. managing client capabilities, user preferences or demographics, processing of multiple end-users preferences to derive collaborative data
- H04N21/25808—Management of client data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/4104—Peripherals receiving signals from specially adapted client devices
- H04N21/4131—Peripherals receiving signals from specially adapted client devices home appliance, e.g. lighting, air conditioning system, metering devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/42203—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] sound input device, e.g. microphone
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
- H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
- H04N21/4222—Remote control device emulator integrated into a non-television apparatus, e.g. a PDA, media center or smart toy
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
- H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
- H04N21/42222—Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
- H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
- H04N21/43615—Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
- H04N21/4394—Processing of audio elementary streams involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/488—Data services, e.g. news ticker
- H04N21/4882—Data services, e.g. news ticker for displaying messages, e.g. warnings, reminders
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/65—Transmission of management data between client and server
- H04N21/654—Transmission by server directed to the client
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/65—Transmission of management data between client and server
- H04N21/658—Transmission by the client directed to the server
- H04N21/6582—Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number
-
- G—PHYSICS
- G08—SIGNALLING
- G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
- G08C2201/00—Transmission systems of control signals via wireless link
- G08C2201/30—User interface
- G08C2201/31—Voice input
-
- G—PHYSICS
- G08—SIGNALLING
- G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
- G08C2201/00—Transmission systems of control signals via wireless link
- G08C2201/40—Remote control systems using repeaters, converters, gateways
-
- G—PHYSICS
- G08—SIGNALLING
- G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
- G08C2201/00—Transmission systems of control signals via wireless link
- G08C2201/90—Additional features
- G08C2201/93—Remote control using other portable devices, e.g. mobile phone, PDA, laptop
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/28—Constructional details of speech recognition systems
- G10L15/30—Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L2015/223—Execution procedure of a spoken command
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
- H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
Definitions
- the present disclosure relates to a method for controlling an electric device whose operation can be controlled using multiple input devices, and such an electric device.
- the techniques disclosed here feature a method for controlling an operation of a target device using a plurality of input devices.
- the method comprises: receiving from one of the plurality of the input devices a first operation instruction issued to the target device, with a first data format; recognizing the first operation instruction and the first data format; determining that the one of the plurality of the input devices is a first input device corresponding to the first data format; and providing to a user of the target device a recommendation for a second input device, a type of the second input device being different from a type of the first input device, when it is determined that a type of the first operation instruction is identical to a type of a second operation instruction received from the second input device earlier than the reception of the first operation instruction.
- the above system can be further improved.
- FIG. 1 is a drawing showing an overview of a speech operation system according to an embodiment
- FIG. 2 is a block diagram showing a speech input device according to the embodiment
- FIG. 3 is a block diagram showing a server according to the embodiment.
- FIG. 4 is a block diagram showing a target device according to the embodiment.
- FIG. 5 is a block diagram showing an input device according to the embodiment.
- FIG. 6 is a diagram showing a sequence in the speech operation system according to the embodiment.
- FIG. 7 is a flowchart showing a device operation determination process according to the embodiment.
- FIG. 8 is a diagram showing a device operation determination table according to the embodiment.
- FIG. 9A is a diagram showing an example of an operation command table according to the embodiment.
- FIG. 9B is a diagram showing an example of an input device ID table according to the embodiment.
- FIG. 10 is a flowchart showing a consecutive operation determination process according to the embodiment.
- FIGS. 11A and 11B are diagrams showing an example of an operation command storage table according to the embodiment.
- FIG. 12 is a flowchart showing a recommended input device presentation determination process according to the embodiment.
- FIG. 13A is a diagram showing an example of a recommended input device determination table according to the embodiment.
- FIG. 13B is a diagram showing an example of a recommended input device ID table according to the embodiment.
- FIG. 14 is a flowchart showing a recommended input device presentation process according to the embodiment.
- FIG. 15A is a diagram showing an example of a presentation method determination table according to the embodiment.
- FIG. 15B is a diagram showing an example of a presentation method ID table according to the embodiment.
- FIG. 16 is a diagram each showing an example of a UI for presenting a recommended input device to a user in the speech operation system according to the embodiment
- FIG. 17 is a diagram each showing an example of a method for presenting the operation method of a recommended input device to a user in the speech operation system according to the embodiment;
- FIG. 18 is a diagrams each showing an example of a method for presenting the position of a recommended input device to a user in the speech operation system according to the embodiment
- FIG. 19 is a flowchart showing an input frequency determination process according to the embodiment.
- FIG. 20 is a flowchart showing a learning process according to the embodiment.
- FIG. 21 is a flowchart showing an operation determination process according to the embodiment.
- FIG. 22 is a flowchart showing an operation determination process according to the embodiment.
- FIG. 23 is a flowchart showing a speech recognition likelihood determination process according to the embodiment.
- FIG. 24 is a flowchart showing a noise level determination process according to the embodiment.
- FIG. 25A is a diagram showing an overview of services provided by an information management system according to the embodiment.
- FIG. 25B is a diagram showing an example in which a device manufacturer serves as a data center operating company according to the embodiment.
- FIG. 25C is a diagram showing an example in which both or one of a device manufacturer and a management company serve as a data center operating company according to the embodiment;
- FIG. 26 is a diagram showing an overview of services provided by an information management system of service type 1 (own data center type cloud services) according to the embodiment;
- FIG. 27 is a diagram showing an overview of services provided by an information management system of service type 2 (IaaS cloud services) according to the embodiment;
- FIG. 28 is a diagram showing an overview of services provided by an information management system of service type 3 (PaaS cloud services) according to the embodiment.
- FIG. 29 is a diagram showing an overview of services provided by an information management system of service type 4 (SaaS cloud services) according to the embodiment.
- SaaS cloud services an information management system of service type 4
- the user can directly give an instruction to the target device by transmitting an operation command to the target device using speech input.
- an operation using speech input may be simpler than an operation performed on an input device such as a remote control to give an instruction to the target device.
- the user can more easily perform the desired operation by using speech input than using a remote control to open the menu window and then select an item corresponding to the desired operation or than using a remote control to input a phrase to be searched for.
- command reception devices capable of receiving both command input using a remote control and command input using a speech (e.g., Japanese Unexamined Patent Application Publication No. 2003-114698).
- Japanese Unexamined Patent Application Publication No. 2003-114698 discloses a control device having a speech recognition function, a switch with which the user externally inputs various types of commands, data, or the like, a display device for displaying images, and a microphone for inputting a speech.
- This control device receives a command, data, or the like inputted by the user from the switch or microphone, processes the received command or the like, and outputs the result of the processing to the display device. If a command inputted using the switch can also be inputted using a speech through the microphone, the control device notifies the user that the command can be inputted using a speech.
- this technology notifies the user that the command can be inputted using a speech. Accordingly, even when the user thinks that the user can more easily perform an operation on the device by using the remote control and then does so, if the command for that operation can be inputted using a speech, the user receives a notification to that effect, thereby feeling annoyed. That is, this background technology cannot issue a notification to the user while distinguishing between an operation suitable for command input using a speech and an operation suitable for command input using the remote control.
- the user can directly give an instruction to the target device by transmitting an operation command to the target device using speech input and therefore such an operation using speech input may be simpler than an operation performed on an input device such as a remote control to give an instruction to the target device.
- the buttons of the remote control correspond to operations, such as power-on/off, the change of the television channel, and the control of the television volume, one-on-one, the user may be able to perform the desired operation more easily by operating the buttons than by using speech input as long as the user remembers the positions of the buttons. Accordingly, if the user can properly select between command input using a speech and command input using the remote control, he or she can more easily perform the desired operation on the target device.
- a method is a method for controlling an operation of a target device using a plurality of input devices.
- the method includes receiving, from one of the plurality of the input devices, a first operation instruction issued to the target device, with a first data format; recognizing the received first operation instruction and the first data format; determining that the one of the plurality of the input devices is a first input device corresponding to the recognized first data format; and providing to a user of the target device a recommendation for a second input device, a type of the second input device being different from a type of the first input device, when it is determined that a type of the first operation instruction is identical to a type of a second operation instruction that is received from the second input device earlier than the reception of the first operation instruction from the first input device.
- a second input device different from the first input device is presented to the user when the user performs an operation which is not suitable for the first input device, it is possible to notify the user that the user can more easily perform the desired operation on the target device. Further, the user can more easily perform the desired operation by using the second input device.
- the first input device may include a speech input device, a remote controller, or a mobile phone
- the first data format may include a data format for a communication using at least one of a speech input device, a remote controller, or a mobile phone
- each of the type of the first operation instruction and the type of the second operation instruction include a control for turning on/off power of the target device, a volume control when the target device is a television set, and an airflow control when the target device is an air conditioner
- the method may further include whether the type of the first operation instruction is identical to the type of the second operation instruction, when a second time when the second operation instruction is received from the second input device falls within a predetermined time period prior to a first time when the first operation instruction is received from the first input device.
- the second input device suitable for the operation can be presented to the user.
- the method may further include determining whether the type of the first operation instruction is identical to the type of the second operation instruction, when the second operation instruction is received from the second input device immediately before a first time when the first operation instruction is received from the first input device. Thus, if the user performs the same operations consecutively, a second input device suitable for the operations can be presented to the user.
- the method may further include storing recommended input device information about the type of the second input device suitable to the type of the first operation instruction; and determining that the type of the first operation instruction is suitable to the type of the second input device based on the recommended input device information.
- a second input device suitable for the operation performed by the user can be presented to the user.
- the target device may be capable of receiving the first operation instruction from the second input device.
- the operation of the target device can be controlled using the multiple input devices.
- the second input device may include at least one of a speech input device, a remote controller, or a mobile phone.
- a sound stating that the second input device may the speech input device, the remote controller, or the mobile phone may be outputted toward the user.
- the user can recognize a second input device suitable for the operation through the sound.
- the target device may be a television set
- the second input device includes a remote controller
- the recommendation for the second input device may be provided to the user by displaying on a display device of the television set an image i) indicating an appearance of the remote controller and ii) highlighting an operation portion capable of performing the first operation instruction, of the remote controller.
- the user can understand which part of the remote control serving as a second input device he or she should operate.
- the method may further include storing time information indicating a time when each of operation instructions has been received; and calculating a receiving frequency of the same type of operation instruction as the first operation instruction in a predetermined period of time, on the basis of the time information, to present to the user the recommendation for the second input device on the basis of the calculated receiving frequency.
- time information indicating a time when each of operation instructions has been received
- calculating a receiving frequency of the same type of operation instruction as the first operation instruction in a predetermined period of time on the basis of the time information, to present to the user the recommendation for the second input device on the basis of the calculated receiving frequency.
- the second input device is provided to the user.
- a second input device suitable for the operation can be presented to the user.
- the method may further include recognizing whether the first input device is the speech input device; recognizing whether the first instruction does not include an operating range for an operation indicated by the first operation instruction; when the first input device is recognized to be the speech input device and when the first instruction does not include the operating range, determining that the speech input device is not suitable to the type of the first operation instruction.
- An electric device is an electric device, operation of the electric device capable to be controlled using a plurality of input devices
- the electric device includes a processor; and a non-transitory memory storing thereon a program, which executed by the processor, causes the processor to: receive, from one of the plurality of the input devices, a first operation instruction issued to the electric device, with a first data format; recognize the received first operation instruction and the first data format; determine that the one of the plurality of the input devices is a first input device corresponding to the recognized first data format; and provide to a user of the electric device a recommendation for a second input device, a type of the second input device being different from a type of the first input device, when it is determined that a type of the first operation instruction is identical to a type of a second operation instruction that is received from the second input device earlier than the reception of the first operation instruction from the first input device.
- a computer program is a non-transitory computer-readable recording medium storing a program for controlling an electric device, operation of the electric device capable to be controlled using a plurality of input devices.
- the computer program causing a computer of the electric device to: receive, from one of the plurality of the input devices, a first operation instruction issued to the electric device, with a first data format; recognize the received first operation instruction and the first data format; determine that the one of the plurality of the input devices is a first input device corresponding to the recognized first data format; and provide to a user of the electric device a recommendation for a second input device, a type of the second input device being different from a type of the first input device, when it is determined that a type of the first operation instruction is identical to a type of a second operation instruction that is received from the second input device earlier than the reception of the first operation instruction from the first input device. Since a second input device different from the first input device is presented to the user when the user performs an operation which is not suitable for the
- a method is a method for controlling an operation of a target device using a plurality of input devices including a speech input device.
- the method includes acquiring, from the speech input device, speech information including I) environmental sound around the speech input device and ii) a speech instruction indicating an operation instruction issued to the target device; calculating a level of noise included in the speech information; recognizing the operation instruction indicated by the speech instruction; recognizing a type of the operation instruction based on the recognition result of the operation instruction; and providing to a user of the target device a recommendation for a second input device on the basis of the calculated noise level and the recognized type of the operation instruction, wherein a type of the second input device does not include speech type. Since a second input device different from the speech input device is presented to the user on the basis of the noise level, it is possible to avoid performing a different operation from the operation intended by the user.
- the recommendation for the second input device may be determined based on i) the calculated noise level, ii) the recognized type of the operation instruction, and iii) recommended input device information, wherein the recommended input device information indicates an input device suitable to each operation instruction type.
- the recommended input device information indicates an input device suitable to each operation instruction type.
- the recommendation for the second input device may be provided to the user.
- the recommendation for the second input device may be provided to the user.
- a method is a method for controlling an operation of a target device using a plurality of input devices including a speech input device
- the method includes acquiring, from the speech input device, speech information including an operation instruction issued to the target device; recognizing the operation instruction included in the speech information; recognizing a type of the operation instruction based on the recognition result of the operation instruction; calculating a likelihood of the recognized operation instruction; and providing to a user of the target device a recommendation for a second input device on the basis of the recognized likelihood and the recognized type of the operation instruction, wherein a type of the second input device does not include speech type.
- the recommendation for the second input device may be determined based on i) the recognized likelihood, ii) the recognized type of the operation instruction, and iii) recommended input device information, wherein the recommended input device information indicates an input device suitable to each operation instruction type.
- a second input device suitable for a condition under which the likelihood of speech input is low can be presented to the user.
- the recommendation for the second input device may be determined based on i) the recognized likelihood, ii) the recognized type of the operation instruction, and iii) recommended input device information, wherein the recommended input device information indicates an input device suitable to each operation instruction type.
- FIG. 25A is a diagram showing an overview of the services provided by the information management system according to the present embodiment.
- the information management system includes a group 600 , a data center operating company 610 , and a service provider 620 .
- the group 600 is, for example, a corporation, organization, or household and may have any size.
- the group 600 includes multiple devices 601 including first and second devices, and a home gateway 602 .
- the devices 601 include devices which can be connected to the Internet (e.g., smartphone, personal computer (PC), and television) and devices which cannot be connected to the Internet by themselves (e.g., lighting system, washer, and refrigerator).
- the devices 601 may include devices which cannot be connected to the Internet by themselves but can be connected thereto through the home gateway 602 . Users 6 use the devices 601 in the group 600 .
- the data center operating company 610 includes a cloud server 611 .
- the cloud server 611 is a virtual server that cooperates with various devices through the Internet.
- the cloud server 611 mostly manages big data or the like, which is difficult to handle using a typical database management tool or the like.
- the data center operating company 610 performs the management of the data, the management of the cloud server 611 , the operation of a data center which performs those, and the like. Details of the operation performed by the data center operating company 610 will be described later.
- the data center operating company 610 is not limited to a corporation which only manages the data or cloud server 611 .
- a device manufacturer which develops or manufactures one of the devices 601 is also performing the management of the data or cloud server 611 , or other operations, the device manufacturer serves as the data center operating company 610 .
- FIG. 25C if a device manufacturer and a management company jointly manage the data or cloud server 611 or share such management, both or either one thereof serve as the data center operating company 610 .
- the service provider 620 includes a server 621 .
- the server 621 may have any size and may be, for example, a memory in a personal computer (PC).
- the service provider 620 need not necessarily include the server 621 .
- the information management system need not necessarily include the home gateway 602 .
- the cloud server 611 manages all the data
- the information management system does not have to include the home gateway 602 .
- any device which cannot be connected to the Internet by itself does not exist, like cases in which all devices in the household are connected to the Internet.
- the first and second devices in the group 600 transmit log information thereof to the cloud server 611 of the data center operating company 610 .
- the cloud server 611 accumulates the log information of the first and second devices (an arrow 631 in FIG. 25A ).
- the log information is information indicating the operating state, operation date/time, or the like of each of the devices 601 .
- the log information is the viewing history of a television, information about the programmed recording of a recorder, the operation date/time of a washer, the amount of laundry, the date/time or frequency of opening/closure of a refrigerator, but is not limited thereto. It may be various types of information available from various types of devices.
- the log information may be directly provided to the cloud server 611 by the devices 601 through the Internet.
- the log information from the devices 601 may be temporarily accumulated in the home gateway 602 and then provided to the cloud server 611 by the home gateway 602 .
- the cloud server 611 of the data center operating company 610 provides a predetermined amount of the accumulated log information to the service provider 620 .
- the predetermined amount may be an amount obtained by compiling the information accumulated in the data center operating company 610 so that the information can be provided to the service provider 620 , or may be an amount requested by the service provider 620 .
- the log information need not necessarily be provided in the predetermined amount, and the amount of the log information to be provided may be changed according to the situation.
- the log information is stored in the server 621 held by the service provider 620 as necessary (an arrow 632 in FIG. 25A ).
- the service provider 620 organizes the log information into information suitable for services to be provided to users and then provides the resulting information to the users.
- the users to which such information is provided may be the users 6 , who use the devices 601 , or may be external users 7 .
- the method for providing the information to the users 6 or 7 may be, for example, to provide the information directly to the users 6 or 7 by the service provider 620 (arrows 633 , 634 in FIG. 25A ).
- the method for providing information to the users 6 may also be to provide the information to the users 6 again through the cloud server 611 of the data center operating company 610 (arrows 635 , 636 in FIG. 25A ). Further, the cloud server 611 of the data center operating company 610 may organize the log information into information suitable for services to be provided to users and then provide the resulting information to the service provider 620 .
- the users 6 may be the same as the external users 7 or may differ therefrom.
- FIG. 1 is a drawing showing an overview of a speech operation system according to the present embodiment.
- the speech operation system includes a speech input device 1 , a server 2 , target devices 3 , input devices 4 , and a network 5 .
- the speech input device 1 acquires a speech generated by the user and transmits the acquired speech to the server 2 .
- the server 2 transmits or receives data to or from the speech input device 1 , target devices 3 , and input devices 4 .
- Each target device 3 performs an operation indicated by a device operation command received from the server 2 and determines whether there is any recommended input device suitable for that operation except for the speech input device.
- Each input device 4 receives input information from the user and transmits the position information of the input device.
- the network 5 connects the speech input device 1 , server 2 , target devices 3 , and input devices 4 to each other using a communication method such as a wired or wireless LAN or Bluetooth®.
- the target devices 3 are electric devices whose operation can be controlled using multiple input devices including the speech input device 1 .
- the target devices 3 include, for example, a television, a recorder, an air-conditioner, a lighting system, an audio system, a telephone, an intercommunication system, and the like.
- one or both of the speech input device 1 and server 2 may be incorporated into each target device 3 .
- FIG. 2 is a block diagram showing an example of the specific elements of the speech input device 1 according to the present embodiment.
- the speech input device 1 includes a speech acquisition unit 101 , a speech detection unit 102 , a speech section cutout unit 103 , a feature value calculation unit 104 , and a communication unit 105 .
- the speech acquisition unit 101 detects a speech.
- the speech detection unit 102 determines whether the speech has been detected.
- the speech section cutout unit 103 detects the section in which the speech inputted by the user exists, of the acquired speech signal and cuts out the detected speech section, since the speech signal includes noise or the like.
- the feature value calculation unit 104 calculates the feature value of the speech on the basis of the cut-out speech section.
- the communication unit 105 transmits the calculated speech feature value to the server 2 .
- the speech input device 1 is, for example, a microphone incorporated in or connected to each target device 3 , a microphone incorporated in a remote control included with each target device 3 or the like, a microphone incorporated in or connected to a mobile communication terminal, or a sound concentrating microphone placed in the house.
- At least some of the elements of the speech input device 1 can be implemented by a microcomputer and a memory.
- the speech detection unit 102 , speech section cutout unit 103 , and feature value calculation unit 104 can be implemented by a microcomputer and a memory.
- the microcomputer performs the above processes on the basis of a computer program read from the memory.
- FIG. 3 is a block diagram showing an example of the specific elements of the server 2 according to the present embodiment.
- the server 2 includes a communication unit 201 , a speech recognition unit 202 , a speech recognition dictionary storage unit 203 , a device operation determination unit 204 , and a device operation determination table 205 .
- the communication unit 201 receives the speech feature value transmitted by the communication unit 105 of the speech input device 1 and transmits an operation command and the ID of the input device to the corresponding target device 3 .
- the speech recognition unit 202 converts the received speech feature value into character strings or word strings on the basis of information in the speech recognition dictionary storage unit 203 .
- the speech recognition dictionary storage unit 203 are storing spoken sentences which may be inputted to the target devices 3 , as shown in FIG. 8 .
- it is storing device operation-related spoken sentences such as “power on” and “increase the volume” and small talk-related spoken sentences such as “good morning” and “tell the weather.” Details of the table shown in FIG. 8 will be described later.
- the device operation determination unit 204 determines an operation command from the result of the speech recognition on the basis of the device operation determination table 205 .
- the device operation determination table 205 is storing operation commands and input device IDs.
- the device operation determination table 205 is storing the following information; the operation command for powering on the television is C 001 ; the operation command for controlling the volume of the television is C 002 ; the ID of the speech input device, to which a speech is inputted, is I 001 , and the ID of the remote control whose buttons are operated is I 002 . Details of the tables shown in FIG. 9A and 9B will be described later.
- At least some of the elements of the server 2 can be implemented by a microcomputer and a memory.
- the speech recognition unit 202 , speech recognition dictionary storage unit 203 , device operation determination unit 204 , and device operation determination table 205 can be implemented by a microcomputer and a memory.
- the microcomputer performs the above processes on the basis of a computer program read from the memory.
- FIG. 4 is a block diagram showing an example of the specific elements of one of the target devices 3 according to the present embodiment.
- the target device 3 includes a communication unit 301 , a device control unit 302 , a consecutive operation determination unit 303 , an operation command storage table 304 , a recommended input device determination unit 305 , a recommended input device determination table 306 , a recommended input device position determination unit 307 , and an operation method display unit 308 .
- the communication unit 301 receives the operation command and input device ID transmitted by the server 2 .
- the device control unit 302 recognizes an operation instruction indicated by the received operation command, as well as recognizes an input device which has received an operation related to the operation instruction from the user, on the basis of the received input device ID.
- the device control unit 302 then controls the operation of the target device 3 in accordance with the received operation command.
- the consecutive operation determination unit 303 determines whether the user has consecutively inputted operations, on the basis of the operation command storage table 304 .
- the operation command storage table 304 is storing histories of operation commands inputted by the user. Details of the history information shown in FIG. 11A will be described later.
- the recommended input device determination unit 305 determines a recommended input device on the basis of the recommended input device determination table 306 .
- the recommended input device determination table 306 is storing operation commands and corresponding recommended input devices. For example, it is storing an operation command C 001 and a corresponding recommended input device ID “UI 001 ,” which represents the speech input device using speech input. Details of the tables shown in FIGS. 13A and 13B will be described later.
- the recommended input device position determination unit 307 determines the position of the recommended input device on the basis of information in a position information acquisition unit 401 and then determines whether to present the user with the position of the recommended input device.
- the operation method display unit 308 presents the user with the recommended input device and the position thereof on the basis of the determinations made by the recommended input device determination unit 305 and recommended input device position determination unit 307 .
- At least some of the elements of the target device 3 can be implemented by a microcomputer and a memory.
- the device control unit 302 , consecutive operation determination unit 303 , operation command storage table 304 , recommended input device determination unit 305 , recommended input device determination table 306 , and recommended input device position determination unit 307 can be implemented by a microcomputer and a memory.
- the microcomputer performs the above processes on the basis of a computer program read from the memory.
- FIG. 5 is a block diagram showing an example of the specific elements of one of the input devices 4 according to the present embodiment.
- the input device 4 includes the position information acquisition unit 401 , an input unit 402 , and a communication unit 403 .
- the position information acquisition unit 401 acquires the position of the input device 4 with respect to the user.
- the input unit 402 acquires information that the user has inputted by operating the buttons or by touching on the smartphone screen.
- the communication unit 403 transmits the information acquired by the position information acquisition unit 401 and input unit 402 to the server 2 or corresponding target device 3 .
- the input device 4 is, for example, one which receives input through the buttons, one which receives input through the touchscreen, or one which receives input through a gesture, but not limited thereto. It may be an input device of any type other than speech input type, as long as the input device can transmit an operation command to the target device 3 .
- At least some of the elements of the input device 4 can be implemented by a microcomputer and a memory.
- the position information acquisition unit 401 and input unit 402 can be implemented by a microcomputer and a memory.
- the microcomputer performs the above processes on the basis of a computer program read from the memory.
- FIG. 6 is a diagram showing a sequence in the speech operation system according to the present embodiment.
- step S 001 the speech input device 1 calculates the feature value of a speech and transmits it to the server 2 .
- the server 2 performs a speech recognition process, that is, it converts the received speech feature value into character strings or word strings on the basis of the information in the speech recognition dictionary of the speech recognition dictionary storage unit 203 .
- step S 003 the server 2 determines the type of a target device and a device operation intended by the user and transmits a corresponding operation command and input device ID to a corresponding target device 3 . Details of the process in step S 003 will be described later.
- step S 004 the target device 3 determines whether it can actually perform the operation command transmitted in step S 003 . For example, if the target device 3 , which is a television, receives an operation command for controlling the volume with the television powered off, it cannot perform the operation.
- step S 005 the target device 3 determines whether the operation performed in step S 004 is the same as the immediately preceding operation. Details of the process in step S 005 will be described later. If the target device 3 determines in step S 005 that operations have been performed consecutively, it performs a process in step S 006 .
- step S 006 the target device 3 determines whether, with respect to the operation instruction inputted by the user, there are any other input devices 4 which are recommended over the speech input device 1 . Details of the process in step S 006 will be described later. If there are any input devices 4 which are recommended over the speech input device 1 , the respective input devices 4 perform a process in step S 007 .
- the recommended input devices 4 determined in step S 006 each determine the position thereof.
- the method for determining the position include the following: if each input device 4 includes a pressure sensor, it determines whether the user is holding the input device 4 , based on whether there is a pressure; if each input device 4 includes a position sensor such as a GPS, RFID, or infrared ID, it determines the position using the position sensor; each input device 4 determines the position using the transmission/reception information of communication radio waves in a wireless LAN, Bluetooth, or the like; if the target device 3 includes a position acquisition unit equivalent to an input device 4 , the relative positions of the input devices 4 and target device 3 are determined; and if the target device 3 includes a camera, the position of the user is estimated using camera information, and the relative positions of the input devices 4 , user, and target device 3 are determined.
- step S 008 a recommended input device which is most suitable for the operation command is shown to the user on the basis of the position information of the recommended input devices determined in step S 007 . Details of the process in step S 008 will be described later.
- FIG. 7 is a flowchart showing the steps of the device operation determination process according to the present embodiment.
- the server 2 determines whether the speech inputted by the user includes an operation instruction issued to a target device 3 , on the basis of the result of the speech recognition.
- step S 301 the server 2 receives the character stings or word strings, which are the result of the speech recognition.
- step S 302 the server 2 determines whether the result of the speech recognition includes an operation to be performed on a target device 3 . Specifically, the server 2 determines whether the result of the speech recognition includes the type of a target device 3 such as a “television” or “air conditioner” or a phrase indicating an operation to be performed on a target device 3 , such as “”power on” or “increase the volume.” If so determined, the process proceeds to step S 303 ; not so determined, the process proceeds to step S 305 .
- a target device 3 such as a “television” or “air conditioner” or a phrase indicating an operation to be performed on a target device 3 , such as “”power on” or “increase the volume.”
- step S 303 the server 2 calls an operation command corresponding to the result of the speech recognition on the basis of the device operation determination table in FIG. 8 .
- step S 304 the server 2 transmits, to the target device 3 , the operation command called in step S 303 , as well as an input device ID recognized based on the input device ID table of FIG. 9B and representing the speech input device.
- step S 305 the speech operation system calls a function other than the target device operation function on the basis of the result of the speech recognition and then performs the called function. For example, if a speech “tell the weather” has been inputted by the user, the speech operation system calls a Q&A function rather than the device operation function. For example, in response to the speech “tell the weather” inputted by the user, the speech operation system makes a response “what area's weather do you want to know?” or a response “today's weather is sunny.”
- FIG. 8 is a diagram showing the device operation determination table for determining a device operation command from the result of speech recognition according to the present embodiment.
- respective speech recognition results and corresponding operation commands are stored in the transverse (row) direction. Note that a device operation determination table may be generated for each target device 3 . These operation commands represent the types of operations inputted by the user.
- an operation command “C 001 ” is called from the device operation determination table of FIG. 8 .
- the device operation determination table multiple speech recognition results may be associated with a single operation command. This is because the user may generate a speech “power on” or may generate a speech “turn on the power” in inputting a speech to issue a command for powering on a target device, that is, even when the user intends the same operation, he or she may use a different speech expression.
- FIG. 9A is a diagram showing an operation command table
- FIG. 9B is a diagram showing an input device ID table.
- respective operation commands and corresponding target device types and device operations are stored in the transverse (row) direction.
- respective input device IDs and corresponding input devices and input methods are stored in the transverse (row) direction.
- an operation command C 001 when a speech “power on” is inputted to the speech input device 1 , an operation command C 001 , as well as an input device ID I 001 are called. Note that there may be operation commands which include fine adjustments such as “increase the volume by 1” and “decrease the volume by 1.”
- FIG. 10 is a flowchart showing the steps of the consecutive operation determination process according to the present embodiment.
- the consecutive operation determination unit 303 of the target device 3 determines whether the user has performed the same operations consecutively.
- step S 501 the consecutive operation determination unit 303 stores the operation command transmitted in step S 304 , in the operation number N 002 of the operation command storage table shown in FIG. 11A .
- step S 502 the continuous operation determination unit 303 determines whether any operation command is stored in the operation number N 001 of the operation command storage table. If so determined, the process proceeds to step S 503 ; if not so determined, the process proceeds to step S 506 .
- step S 503 If the respective operation commands stored in the operation numbers N 001 and N 002 are matched in step S 503 , the process proceeds to step S 504 . If the operation commands stored in the operation numbers N 001 and N 002 are not matched, the process proceeds to step S 506 .
- step S 504 the consecutive operation determination unit 303 confirms that the operations are consecutive and then discards the operation command information stored in the operation numbers N 001 and N 002 .
- the process proceeds to step S 505 .
- step S 505 the recommended input device determination unit 305 performs a recommended input device presentation determination process, ending the consecutive operation determination process.
- step S 506 If the consecutive operation determination unit 303 determines in step S 502 or S 503 that the operations are not consecutive, it performs step S 506 . It stores the operation command stored in the operation number N 002 , in the operation number N 001 .
- the recommended input device is presented to the user.
- a message such as “use of the remote control is recommended in order to control the volume” does not appear on the screen. Accordingly, such a message is prevented from hindering the viewing of the television by the user. That is, such a message is prevented from appearing each time the user performs an operation using an input device other than the recommended input device, thereby reducing the annoyance of the user.
- the user can learn selective use of an input device for each operation command.
- step S 501 For example, if the user inputs a speech “power on” as the first operation, an operation command C 001 is stored in the operation number N 002 in step S 501 . Then, in step S 502 , it is determined that no operation command is stored in the operation number N 001 . Accordingly, the process proceeds to step S 506 to store the operation command C 001 stored in the operation number N 002 , in the operation number N 001 .
- step S 502 C 001 is stored in the operation number N 001 . Accordingly, the process proceeds to step S 503 .
- step S 503 the operation numbers N 001 and N 002 are not matched, since C 001 is stored in the operation number N 001 and C 002 is stored in the operation number N 002 . Accordingly, the process proceeds to step S 506 to store C 002 stored in the operation number N 002 , in the operation number N 001 .
- step S 503 the operation numbers N 001 and N 002 are matched, since C 002 is stored in the operation number N 001 and C 002 is stored in the operation number N 002 . Accordingly, the process proceeds to step S 504 . In step S 504 , C 002 stored in both the operation numbers N 001 and N 002 is discarded. The process proceeds to S 505 to perform a recommended input device presentation determination process. Thus, the consecutive operation determination process ends.
- FIG. 11A is a diagram showing an example of the operation command storage table according to the present embodiment.
- the operation command storage table is storing history information indicating an operation command received earlier than the current operation command. Thus, the types of the operations inputted by the user can be recognized based on the stored operation commands.
- the operation command storage table is used in the consecutive operation determination process.
- respective operation numbers and corresponding operation commands are stored in the transverse (row) direction. If the respective operation commands stored in the operation numbers N 001 and N 002 are matched in step S 503 , those operation commands are discarded. In contrast, if the operation commands stored in the operation numbers N 001 and N 002 are not matched in step S 503 , the operation command stored in the operation number N 002 is stored in the operation number N 001 .
- the operation command storage table may include time information indicating the time when each operation command has been inputted.
- FIG. 11B is a diagram showing an operation command storage table including such time information. If the operation command storage table includes time information, operation commands to be compared may be the latest operation command and an operation command which has been inputted within a predetermined period of time before the latest operation command has been inputted. Operation commands to be compared may also be the latest operation command and an operation command which has been inputted immediately before the latest operation command has been inputted. While the examples in which two operation numbers are stored are shown in FIGS. 11A and 11B , three or more operation numbers may be stored.
- FIG. 12 is a flowchart showing the steps of the recommended input device presentation determination process according to the present embodiment.
- the recommended input device determination unit 305 determines whether any input device other than the speech input device is set as a recommended input device.
- step S 601 the recommended input device determination unit 305 receives the input device ID transmitted in step S 304 .
- step S 602 the recommended input device determination unit 305 determines whether the input device ID is I 001 representing the speech input device to determine whether the user has operated the target device using speech input. If so determined, the process proceeds to step S 603 ; if not so determined, the process ends.
- step S 603 the recommended input device determination unit 305 refers to information in the recommended input device determination table shown in FIG. 13A and recommended input device ID information shown in FIG. 13B to determine whether a recommended input device ID corresponding to the operation command of the device operation performed by the user represents the speech input device or any other input device.
- the recommended input device determination unit 305 can determine that it is appropriate that the user has inputted the speech to the speech input device to operate the device and that there is no need to recommend any other input device to the user. Accordingly, the recommended input device determination unit 305 ends the recommended input device presentation determination process without performing a recommended input device position determination process.
- step S 604 the recommended input device determination unit 305 determines that with respect to the device operation performed by the user, there is an input device other than the speech input device, and the recommended input device position determination unit 307 performs a recommended input device position determination process.
- the recommended input device determination unit 305 receives the input device ID I 001 representing the speech input device in step S 304 each time.
- the received input device ID is I 001 . Accordingly, the process proceeds to step S 603 .
- an operation using the remote control or touchscreen is recommended over an operation using a speech with respect to C 002 on the basis of the recommended input device determination table shown in FIG. 13A . Accordingly, the process proceeds to step S 604 to perform a recommended input device position determination process.
- the user operates the television using multiple input devices such as the speech input device, remote control, and smartphone. While the user can more easily input a keyword by using speech input when searching for a program, the user can more easily operate the television by using a fast-feedback device such as the remote control when controlling the volume as desired by the user.
- a fast-feedback device such as the remote control
- he or she inputs a speech “increase the volume” or a speech “decrease the volume” a number of times in an attempt to control the volume using speech input. Thus, the user feels annoyed about speech input.
- the recommended input device presentation determination process is performed.
- the recommended input devices are presented to the user.
- the user can know whether the inputted operation command is suitable for speech input.
- the user can learn selective use of operation commands suitable for speech input and other input devices and thus can selectively use the recommended multiple input devices and reduce the annoyance associated with use of the multiple input devices.
- FIG. 13A is a diagram showing an example of the recommended input device determination table
- FIG. 13B is a diagram showing an example of the recommended input device ID table.
- respective operation commands and corresponding recommended input device IDs are stored in the transverse (row) direction.
- respective recommended input device IDs and corresponding input devices and input methods are stored in the transverse (row) direction.
- FIG. 14 is a flowchart showing the steps of the recommended input device presentation process according to the present embodiment.
- the operation method display unit 308 presents, to the user, a recommended input device closer to the user on the basis of the recommended input device and the position information of the recommended input device.
- step S 801 the operation method display unit 308 receives the recommended input device IDs and recommended input device position information transmitted in steps S 006 and S 007 , respectively. If it has received no recommended input device IDs or recommended input device position information, the operation method display unit 308 repeats step S 801 until it receives them. If the operation method display unit 308 determines in step S 802 that a single recommended input device ID has been received in step S 801 , the process proceeds to step S 804 . If the operation method display unit 308 determines in step S 802 that multiple recommended input device IDs have been received in step S 801 , the process proceeds to step S 804 .
- step S 803 the operation method display unit 308 determines a recommended input device which is closer to the user, of the multiple recommended input device IDs and decides the determined recommended input device as a recommended input device to be presented to the user. Note that all the multiple recommended input devices may be presented to the user without performing steps S 802 and S 803 .
- step S 804 the operation method display unit 308 presents the recommended input device other than the speech input device to the user, for example, using a method shown in FIG. 16 or 17 . Details of FIGS. 16 and 17 will be described later.
- step S 805 the operation method display unit 308 determines a method for presenting the position information of the recommended input device to the user on the basis of information as to whether to present the position of the recommended input device and the presentation method information corresponding to the distance between the user and recommended input device described in a table shown in FIG. 15A . If the distance between the user and recommended input device is shortest, the user may be holding the recommended input device, or the recommended input device may be within the reach of the user. Accordingly, the position information of the recommended input device is not presented. In contrast, if the distance between the user and recommended input device is long, the process proceeds to step S 806 . The operation method display unit 308 presents the position information of the recommended input device to the user, as shown in FIG.
- the presentation device or presentation method is not limited to the remote control, television, speech, or screen display and can take various forms.
- the operation method display unit 308 receives the respective recommended input device IDs and the position information thereof in step S 801 .
- the number of recommended input device IDs, which correspond to the remote control and smartphone, is two. Accordingly, the process proceeds to step S 803 . If the distance between the remote control and user is 50 cm and the distance between the smartphone and user is 80 cm, the operation method display unit 308 determines in step S 803 that the remote control is closer to the user than the smartphone.
- the operation method display unit 308 presents recommended input device information such as “remote control is recommended” to the user on the basis of the determination in step S 803 .
- the remote control is distant from the user by 50 cm. Accordingly, the process proceeds to step S 806 .
- the operation method display unit 308 presents recommended input device position information to the user by causing the remote control to emit a sound.
- the operation method display unit 308 presents, to the user, a recommended input device which is closest to the user, of the recommended input devices on the basis of the positions of the user and recommended input devices.
- the user can know the position of the recommended input device which is the closest to the user.
- presenting a recommended input device to the user can assist the user in selectively using the speech input device and the multiple input devices including the remote control.
- FIG. 15A is a diagram showing an example of a presentation method determination table
- FIG. 15B is a diagram showing an example of a presentation method ID table.
- the respective distances D between the user and recommended input device and corresponding information as to whether to present the position information of the recommended input device and presentation method IDs are stored in the transverse (row) direction. Note that while the information as to whether to present the position information and the presentation method are changed at 0.3 m and 1.0 m in FIG. 15A , these distances are not limiting. Further, while the range of the distance between the user and recommended input device are divided into three ranges and information as to whether to present the position information and the presentation method vary among the three ranges in FIG.
- the range of the distance between the user and recommended input device is divided into two or four or more ranges and information as to whether to present the position information and the presentation method vary among such ranges.
- the presentation method determination table respective presentation method IDs and corresponding presentation devices and presentation methods are stored in the transverse (row) direction.
- the stored presentation devices and presentation methods are, for example, devices and methods shown in FIGS. 16, 17 and 18A to 18C .
- FIG. 16 is a drawing showing an example of a user interface (UI) for presenting a recommended input device to the user in the speech operation system according to the present embodiment.
- UI user interface
- FIG. 16 show examples of a UI which when there is a recommended input device that is more suitable for inputting an device operation than speech input, presents such a recommended input device to the user.
- the user who has inputted speeches consecutively to change the volume of the television as the target device 3 is being recommended to operate the television using the remote control.
- a display screen stating “remote control operation is recommended” may be displayed on the television.
- a sound stating “remote control operation is recommended” may be emitted from a speaker included in the television or remote control.
- Yet another presentation method may be to present the user with new information indicating that there is another recommended input device rather than presenting specific information indicating that “remote control operation is recommended” and then to present the user with detailed information only when the user needs it.
- FIG. 17 is a drawing showing an example of a method for specifically presenting the user with the operation method of a recommended input device in the speech operation system according to the present embodiment.
- These drawings show examples of a method for, when there is a recommended input device that is more suitable for inputting a device operation than speech input, specifically presenting the user with the operation method of the recommended input device.
- the user who has inputted speeches consecutively to change the volume of the television is recommended to operate the buttons of the remote control.
- the volume control buttons of the remote control may be allowed to light up.
- another presentation method may be to display a remote control image on the television screen and to highlight portions corresponding to the volume control buttons in the displayed remote control image.
- the user can understand which buttons of the remote control should be pressed to perform the function which the user has performed using speech input.
- FIG. 18 is a drawing showing an example of a method for presenting the user with the position of a recommended input device in the speech operation system according to the present embodiment.
- These drawings show examples of a method for, when there is a recommended input device that is more suitable for inputting a device operation than speech input, presenting the user with the position of the recommended input device.
- the user may be informed of the position of the remote control by lighting the remote control itself.
- the user may be informed of the position of the remote control by causing the remote control to emit a sound.
- the position of the remote control may be presented by displaying a sketch of the room on the television screen. Note that instead of presenting the precise position of the remote control, the range in which the remote control is more likely to be placed, the direction in which the remote control is placed with respect to the television or user, or the like may be presented.
- the speech input device 1 performs the feature value extraction process S 001 on the speech signal issued by the user.
- the server 2 converts the extracted speech feature value into character strings or word strings on the basis of the information in the speech recognition dictionary storage unit 203 .
- the server 2 then performs the device operation determination process S 003 on the basis of the resulting character strings or word strings.
- the server 2 determines that the speech issued by the user intends a “volume control” operation.
- the target device 3 performs the device operation process S 004 on the basis of the determination to increase the volume of the television.
- the target device 3 then performs the consecutive operation determination process S 005 . Specifically, the target device 3 determines whether volume control operation commands have been performed consecutively, determines that the volume control operation command is the first one, and stores the operation command in the operation command storage table 304 , ending the process.
- the speech input device 1 , server 2 , and target device 3 perform similar processes to those which have been performed on the first speech until the consecutive operation determination process S 005 . Since the volume control operation command is stored in the operation number N 001 of the operation command storage table 304 , the target device 3 determines that the user has inputted the two volume control operation commands consecutively. That is, it determines that the user has inputted the same operation commands consecutively. Accordingly, the target device 3 performs the recommended input device presentation determination process S 006 . Specifically, it determines whether there are any recommended input devices other than the speech input device in controlling the volume.
- the target device 3 determines that the remote control or smartphone is more suitable for controlling the volume than speech input, and the input devices 4 perform the recommended input device position determination process S 007 .
- the input devices 4 calculate the distance between the user and remote control, for example, as 25 cm and the distance between the user and smartphone, for example, as 50 cm.
- the target device 3 determines that the recommended input device to be presented to the user is the remote control.
- the target device 3 then performs the recommended input device presentation process S 008 on the basis of the determination. Specifically, it causes the television to emit a sound stating “remote control operation is recommended” toward the user.
- the recommended input device By presenting the recommended input device in this manner, it is possible to present a fast-feedback input device such as the remote control to the user with respect to a troublesome operation such as volume control, in which the user must input an speech “increase the volume” or a speech “decrease the volume” to the speech input device a number of times.
- a fast-feedback input device such as the remote control
- volume control a troublesome operation
- the user can learn that the remote control or smartphone is recommended in controlling the volume.
- Such learning finally allows the user to select a recommended input device for each operation by himself or herself.
- the target device 3 when the user has inputted the same operation instructions consecutively using speeches, the target device 3 performs the recommended input device presentation determination process.
- the target device 3 performs the recommended input device presentation determination process.
- FIG. 19 is a flowchart showing the steps of an input frequency determination process.
- the target device 3 calculates the frequency with which the same type of operation command as the current operation command has been inputted within a predetermined period of time and, if the calculated input frequency is greater than or equal to a predetermined value, presents a recommended input device to the user.
- step S 901 the target device 3 stores the operation command transmitted in step S 304 in an operation command storage table (history information) including time information as shown in FIG. 11B .
- step S 902 the consecutive operation determination unit 303 calculates the frequency with which the same type of operation command as the current operation command has been inputted within the predetermined period of time (input frequency).
- the predetermined period of time may be any period of time and is set to, for example, 10 sec, 1 min, 30 min, or 1 h in accordance with the type of the target device. In the example of FIG. 11B , the same type of two operation commands have been inputted within one min.
- step S 903 the consecutive operation determination unit 303 determines whether the input frequency is greater than or equal to a predetermined value.
- the predetermined value may be any value and is set to, for example, two. If the input frequency is greater than or equal to the predetermined value, the recommended input device determination unit 305 performs the recommended input device presentation determination process in step S 904 , and the operation method display unit 308 presents a recommended input device to the user on the basis of the determination. For example, the operation method display unit 308 presents the remote control or smartphone, which is an input device other than the speech input device, to the user as a recommended input device. If the input frequency is not greater than or equal to the predetermined value, the recommended input device determination unit 305 ends the process without performing the recommended input device presentation determination process.
- the user has controlled the volume of the television using speech input a number of times within a short period of time, it is possible to present a fast-feedback input device such as the remote control to the user with respect to the troublesome operation using speech input.
- a fast-feedback input device such as the remote control
- the user can learn that the remote control or smartphone is recommended in controlling the volume. Such learning finally allows the user to select a recommended input device for each operation by himself or herself.
- FIG. 20 is a flowchart showing the steps of a learning process.
- the target device 3 manages, as history information, the operation thereof which has changed in accordance with an operation command.
- step S 1001 the communication unit 301 of the target device 3 receives an operation command.
- step S 1002 the device control unit 302 controls the operation of the target device 3 .
- the device control unit 302 stores, as history information, the operation in which the television volume of 15 has been changed to 25.
- the operation may be stored in the operation command storage table 304 serving as history information or may be stored in history information which is different from the operation command storage table 304 .
- the device control unit 302 sets the volume of the television to 25.
- the device control unit 302 then stores, as history information, the operation in which the volume of the television has been changed to 25. By accumulating operations in this manner, it is possible to obtain a learning result indicating that “when increasing the volume, the volume is often changed to 25.” By controlling the device on the basis of such a learning result, it is possible to set the device in accordance with the preferences of the user. Further, since the history information includes specific numerals such as the amount of variation of volume, it is possible to accurately set the device in accordance with the preferences of the user.
- FIG. 21 is a flowchart showing the steps of a process of determining operations corresponding to an operation command.
- step S 1101 the communication unit 301 receives an operation command.
- step S 1102 the device control unit 302 determines whether it can operate the device in accordance with the operation command, which is based on speech input. For example, if a speech stating “increase the volume by 10” has been inputted, the device control unit 302 can operate the device, since the amount of volume to be increased is obvious. Accordingly, in step S 1105 , the device control unit 302 increases the volume of the television by 10.
- the device control unit 302 determines that it cannot operate the device, since the amount of volume to be increased is not obvious. Accordingly, the process proceeds to step S 1103 .
- step S 1103 the device control unit 302 determines whether history information of operations corresponding to commands indicating “increase the volume” is accumulated. For example, the device control unit 302 determines whether corresponding multiple operations are accumulated. If so determined, the device control unit 302 operates the device in accordance with the operations in step S 1105 . For example, if operations in which the volume has been changed to 25 are accumulated, the device control unit 302 sets the volume of the television to 25.
- the recommended input device determination unit 305 performs a recommended input device presentation determination process in step S 1104 , and the operation method display unit 308 presents a recommended input device to the user on the basis of the determination.
- the recommended input device presented is, for example, the remote control or smartphone, which is an input device other than the speech input device.
- FIG. 22 is a flowchart showing another example of a process of determining an operation corresponding to an operation command.
- step S 1201 the communication unit 301 receives an operation command.
- step S 1202 the device control unit 302 determines whether the operation command, which is based on speech input, includes numerical information.
- the numerical information represents a specific numeral indicating the amount of variation by which the operation of the target device is changed. For example, if a speech stating “increase the volume by 10” has been inputted, the corresponding operation command includes the numeral of 10, by which the volume is to be increased. Accordingly, in step S 1205 , the device control unit 302 increases the volume of the television by 10.
- step S 1203 the device control unit 302 determines whether numerical history information satisfying a condition is accumulated. For example, if there is a condition that the current volume of the television is 15 and if history information that the television volume of 15 has been changed to 25 by increasing it by 10 is accumulated, the device control unit 302 sets the volume of the television to 25 by increasing it by 10 in step S 1205 .
- the device control unit 302 determines that any history information satisfying the condition is not accumulated, the recommended input device determination unit 305 performs a recommended input device presentation determination process in step S 1204 , and the operation method display unit 308 presents a recommended input device to the user on the basis of the determination.
- the user who may control the volume using speech input a number of times, can avoid performing such troublesome operations.
- the recommended input device presented is, for example, the remote control or smartphone, which is an input device other than the speech input device.
- FIG. 23 is a flowchart showing the steps of a speech recognition likelihood determination process.
- an operation instruction is recognized from speech information; the likelihood of the recognized operation instruction is calculated; if the likelihood is lower than a predetermined value, a recommended input device suitable for the corresponding operation command is presented to the user.
- step S 1301 the speech recognition unit 202 converts a received speech feature value into character strings or word strings.
- step S 1302 the device operation determination unit 204 recognizes an operation command on the basis of the result of the speech recognition. Note that if the target device 3 has the functions of the speech recognition unit 202 and device operation determination unit 204 , it performs these processes.
- the device control unit 302 calculates the likelihood of the speech recognition.
- the likelihood can be obtained by calculating the distance between the recognized speech and a language model serving as a reference. In this case, the likelihood is higher as the calculated distance is shorter.
- step S 1304 the device control unit 302 determines whether the calculated likelihood is lower than a predetermined value.
- the predetermined value may be any value. If the likelihood is higher than or equal to the predetermined value, the device control unit 302 controls the device in accordance with the recognized operation command in step S 1306 .
- the recommended input device determination unit 305 performs a recommended input device presentation determination process in step S 1305 , and the operation method display unit 308 presents a recommended input device to the user on the basis of the determination.
- the recommended input device presented is, for example, the remote control or smartphone, which is an input device other than the speech input device. If the likelihood is low, an operation based on the inputted speech is more likely to be different from the operation intended by the user. In this case, by presenting a recommended input device, it is possible to avoid performing the operation different from the operation intended by the user.
- FIG. 24 is a flowchart showing the steps of a noise level determination process.
- the noise level of speech information is calculated, and if the noise level is higher than or equal to a predetermined value, a recommended input device suitable for the corresponding operation command is presented to the user.
- step S 1401 the speech recognition unit 202 acquires speech information including a speech feature value.
- This speech information includes the environmental sound around the speech input device and an operation instruction issued to the target device.
- step S 1402 the speech recognition unit 202 calculates the noise level of the speech information.
- the noise level can be calculated by obtaining S/N by making comparison between the sound pressures of the noise and speech. In this case, the noise level is higher as S/N is lower.
- step S 1403 the speech recognition unit 202 determines whether the calculated noise level is higher than or equal to a predetermined value.
- the predetermined value may be any value. If the noise level is lower than the predetermined value, the device control unit 302 controls the target device in accordance with an operation command recognized from the speech information in step S 1405 .
- the recommended input device determination unit 305 performs a recommended input device presentation determination process in step S 1404 , and the operation method display unit 308 presents a recommended input device to the user on the basis of the determination.
- the recommended input device presented is, for example, the remote control or smartphone, which is an input device other than the speech input device. If the noise level is high, an operation based on the inputted speech is more likely to be different from the operation intended by the user. In this case, by presenting a recommended input device, it is possible to avoid performing the operation different from the operation intended by the user.
- a recommended input device presentation determination process may be performed based on both the noise level and likelihood.
- a computer program for causing hardware or the like to perform those processes may be stored in a memory and then executed by a microcomputer.
- Such a computer program may be installed from a recording medium (semiconductor memory, optical disk, etc.) storing the computer program to the respective devices or may be downloaded through an electrical communication line such as the Internet.
- Such a computer program may also be wirelessly installed into the respective devices.
- the technology described in the above aspect can be implemented in the following types of cloud services.
- these types of cloud services are only illustrative.
- FIG. 26 is a diagram showing an overview of services provided by an information management system of service type 1 (own data center type cloud services).
- a service provider 620 acquires information from a group 600 and provides services to a user.
- the service provider 620 has the functions of a data center operating company. That is, the service provider 620 holds a cloud server which manage big data. Accordingly, there is no data center operating company.
- the service provider 620 operates and manages a data center (cloud server) 703 .
- the service provider 620 also manages an operating system (OS) 702 and an application 701 .
- the service provider 620 provides services using the OS 702 and application 701 (arrow 704 ).
- FIG. 27 is a diagram showing an overview of services provided by an information management system of service type 2 (laaS cloud services).
- laaS is the abbreviation of infrastructure as a service and is a cloud service providing model which provides an infrastructure for constructing and operating a computer system, as a service through the Internet.
- a data center operating company 610 operates and manages a data center (cloud server) 703 .
- a service provider 620 manages an OS 702 and an application 701 .
- the service provider 620 provides services using the OS 702 and application 701 (arrow 704 ).
- FIG. 28 is a diagram showing an overview of services provided by an information management system of service type 3 (PaaS cloud services).
- PaaS is the abbreviation of platform as a service and is a cloud service providing model which provides a platform serving as an infrastructure for constructing and operating software, as a service through the Internet.
- a data center operating company 610 manages an OS 702 and operates and manages a data center (cloud server) 703 .
- a service provider 620 manages an application 701 .
- the service provider 620 provides services using the OS 702 and application 701 (arrow 704 ).
- Service Type 4 SaaS Cloud Services
- FIG. 29 is a diagram showing an overview of services provided by an information management system of service type 4 (SaaS cloud services).
- SaaS is the abbreviation of software as a service.
- SaaS cloud services are, for example, a cloud service providing model which allows users not holding a data center (cloud server) such as companies or individuals to use an application provided by a platform provider holding a data center (cloud server) through a network such as the Internet.
- a data center operating company 610 manages an application 701 and an OS 702 and operates and manages a data center (cloud server) 703 .
- a service provider 620 provides services using the OS 702 and application 701 (arrow 704 ).
- the service provider 620 provides services.
- the service provider or data center operating company may develop an OS, application, a database of big data, or the like on its own or may outsource such development to a third party.
- the technology of the present disclosure is particularly useful in the field of technologies that control the operation of a device using a speech.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Computer Graphics (AREA)
- Automation & Control Theory (AREA)
- Acoustics & Sound (AREA)
- Selective Calling Equipment (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
A method is provided for controlling an operation of a target device using a plurality of input devices, including a speech input device. The method includes acquiring, from the speech input device, speech information, including i) environmental sound around the speech input device, and ii) a speech instruction indicating an operation to be performed on the target device. The method also includes calculating a level of noise included in the speech information, and recognizing the operation. The method further includes informing a user of a second input device as a recommended input device based on the calculated noise level and the recognized operation instruction, wherein the second input device does not include a speech input device.
Description
- The present application is a continuation application of U.S. application Ser. No. 14/743,704, filed Jun. 18, 2005, which claims the benefit of Japanese Patent Application Nos. 2015-050967, filed Mar. 13, 2015 and 2014-135899, filed Jul. 1, 2014. The entire disclosure of each of the above-identified application, including the specification, drawings, and claims is incorporated herein by reference in its entirety.
- The present disclosure relates to a method for controlling an electric device whose operation can be controlled using multiple input devices, and such an electric device.
- With the development of speech recognition technology, speech recognition accuracy has been significantly improved in recent years. Accordingly, there have been considered device management systems in which various types of devices are operated using speeches. It is expected that the user will be able to control the various types of devices by uttering desired operations toward the devices, without having to perform troublesome button operations.
- However, such systems still have many matters to be considered and have to be further improved for commercialization.
- In one general aspect, the techniques disclosed here feature a method for controlling an operation of a target device using a plurality of input devices. The method comprises: receiving from one of the plurality of the input devices a first operation instruction issued to the target device, with a first data format; recognizing the first operation instruction and the first data format; determining that the one of the plurality of the input devices is a first input device corresponding to the first data format; and providing to a user of the target device a recommendation for a second input device, a type of the second input device being different from a type of the first input device, when it is determined that a type of the first operation instruction is identical to a type of a second operation instruction received from the second input device earlier than the reception of the first operation instruction.
- These general and specific aspects may be implemented using a system, a method, and a computer program, and any combination of systems, methods, and computer programs.
- According to the above aspect, the above system can be further improved.
- It should be noted that general or specific embodiments may be implemented as a system, a method, an integrated circuit, a computer program, a storage medium, or any selective combination thereof.
- Additional benefits and advantages of the disclosed embodiments will become apparent from the specification and drawings. The benefits and/or advantages may be individually obtained by the various embodiments and features of the specification and drawings, which need not all be provided in order to obtain one or more of such benefits and/or advantages.
-
FIG. 1 is a drawing showing an overview of a speech operation system according to an embodiment; -
FIG. 2 is a block diagram showing a speech input device according to the embodiment; -
FIG. 3 is a block diagram showing a server according to the embodiment; -
FIG. 4 is a block diagram showing a target device according to the embodiment; -
FIG. 5 is a block diagram showing an input device according to the embodiment; -
FIG. 6 is a diagram showing a sequence in the speech operation system according to the embodiment; -
FIG. 7 is a flowchart showing a device operation determination process according to the embodiment; -
FIG. 8 is a diagram showing a device operation determination table according to the embodiment; -
FIG. 9A is a diagram showing an example of an operation command table according to the embodiment; -
FIG. 9B is a diagram showing an example of an input device ID table according to the embodiment; -
FIG. 10 is a flowchart showing a consecutive operation determination process according to the embodiment; -
FIGS. 11A and 11B are diagrams showing an example of an operation command storage table according to the embodiment; -
FIG. 12 is a flowchart showing a recommended input device presentation determination process according to the embodiment; -
FIG. 13A is a diagram showing an example of a recommended input device determination table according to the embodiment; -
FIG. 13B is a diagram showing an example of a recommended input device ID table according to the embodiment; -
FIG. 14 is a flowchart showing a recommended input device presentation process according to the embodiment; -
FIG. 15A is a diagram showing an example of a presentation method determination table according to the embodiment; -
FIG. 15B is a diagram showing an example of a presentation method ID table according to the embodiment; -
FIG. 16 is a diagram each showing an example of a UI for presenting a recommended input device to a user in the speech operation system according to the embodiment; -
FIG. 17 is a diagram each showing an example of a method for presenting the operation method of a recommended input device to a user in the speech operation system according to the embodiment; -
FIG. 18 is a diagrams each showing an example of a method for presenting the position of a recommended input device to a user in the speech operation system according to the embodiment; -
FIG. 19 is a flowchart showing an input frequency determination process according to the embodiment; -
FIG. 20 is a flowchart showing a learning process according to the embodiment; -
FIG. 21 is a flowchart showing an operation determination process according to the embodiment; -
FIG. 22 is a flowchart showing an operation determination process according to the embodiment; -
FIG. 23 is a flowchart showing a speech recognition likelihood determination process according to the embodiment; -
FIG. 24 is a flowchart showing a noise level determination process according to the embodiment; -
FIG. 25A is a diagram showing an overview of services provided by an information management system according to the embodiment; -
FIG. 25B is a diagram showing an example in which a device manufacturer serves as a data center operating company according to the embodiment; -
FIG. 25C is a diagram showing an example in which both or one of a device manufacturer and a management company serve as a data center operating company according to the embodiment; -
FIG. 26 is a diagram showing an overview of services provided by an information management system of service type 1 (own data center type cloud services) according to the embodiment; -
FIG. 27 is a diagram showing an overview of services provided by an information management system of service type 2 (IaaS cloud services) according to the embodiment; -
FIG. 28 is a diagram showing an overview of services provided by an information management system of service type 3 (PaaS cloud services) according to the embodiment; and -
FIG. 29 is a diagram showing an overview of services provided by an information management system of service type 4 (SaaS cloud services) according to the embodiment. - The user can directly give an instruction to the target device by transmitting an operation command to the target device using speech input. Thus, such an operation using speech input may be simpler than an operation performed on an input device such as a remote control to give an instruction to the target device. For example, the user can more easily perform the desired operation by using speech input than using a remote control to open the menu window and then select an item corresponding to the desired operation or than using a remote control to input a phrase to be searched for.
- There are command reception devices capable of receiving both command input using a remote control and command input using a speech (e.g., Japanese Unexamined Patent Application Publication No. 2003-114698).
- Japanese Unexamined Patent Application Publication No. 2003-114698 discloses a control device having a speech recognition function, a switch with which the user externally inputs various types of commands, data, or the like, a display device for displaying images, and a microphone for inputting a speech. This control device receives a command, data, or the like inputted by the user from the switch or microphone, processes the received command or the like, and outputs the result of the processing to the display device. If a command inputted using the switch can also be inputted using a speech through the microphone, the control device notifies the user that the command can be inputted using a speech.
- However, each time a command which can be inputted using a speech is inputted using the switch, this technology notifies the user that the command can be inputted using a speech. Accordingly, even when the user thinks that the user can more easily perform an operation on the device by using the remote control and then does so, if the command for that operation can be inputted using a speech, the user receives a notification to that effect, thereby feeling annoyed. That is, this background technology cannot issue a notification to the user while distinguishing between an operation suitable for command input using a speech and an operation suitable for command input using the remote control.
- As described above, the user can directly give an instruction to the target device by transmitting an operation command to the target device using speech input and therefore such an operation using speech input may be simpler than an operation performed on an input device such as a remote control to give an instruction to the target device. On the other hand, if the buttons of the remote control correspond to operations, such as power-on/off, the change of the television channel, and the control of the television volume, one-on-one, the user may be able to perform the desired operation more easily by operating the buttons than by using speech input as long as the user remembers the positions of the buttons. Accordingly, if the user can properly select between command input using a speech and command input using the remote control, he or she can more easily perform the desired operation on the target device.
- In view of the foregoing, the inventors have conceived of the following modifications to improve the functions of a speech device operation system.
- A method according to one embodiment of the present invention is a method for controlling an operation of a target device using a plurality of input devices. The method includes receiving, from one of the plurality of the input devices, a first operation instruction issued to the target device, with a first data format; recognizing the received first operation instruction and the first data format; determining that the one of the plurality of the input devices is a first input device corresponding to the recognized first data format; and providing to a user of the target device a recommendation for a second input device, a type of the second input device being different from a type of the first input device, when it is determined that a type of the first operation instruction is identical to a type of a second operation instruction that is received from the second input device earlier than the reception of the first operation instruction from the first input device. Since a second input device different from the first input device is presented to the user when the user performs an operation which is not suitable for the first input device, it is possible to notify the user that the user can more easily perform the desired operation on the target device. Further, the user can more easily perform the desired operation by using the second input device.
- In one embodiment, for example, the first input device may include a speech input device, a remote controller, or a mobile phone, the first data format may include a data format for a communication using at least one of a speech input device, a remote controller, or a mobile phone, each of the type of the first operation instruction and the type of the second operation instruction include a control for turning on/off power of the target device, a volume control when the target device is a television set, and an airflow control when the target device is an air conditioner
- In one embodiment, for example, the method may further include whether the type of the first operation instruction is identical to the type of the second operation instruction, when a second time when the second operation instruction is received from the second input device falls within a predetermined time period prior to a first time when the first operation instruction is received from the first input device. Thus, if the user performs the same operation multiple times within a predetermined period of time, the second input device suitable for the operation can be presented to the user.
- In one embodiment, for example, the method may further include determining whether the type of the first operation instruction is identical to the type of the second operation instruction, when the second operation instruction is received from the second input device immediately before a first time when the first operation instruction is received from the first input device. Thus, if the user performs the same operations consecutively, a second input device suitable for the operations can be presented to the user.
- In one embodiment, for example, the method may further include storing recommended input device information about the type of the second input device suitable to the type of the first operation instruction; and determining that the type of the first operation instruction is suitable to the type of the second input device based on the recommended input device information. Thus, a second input device suitable for the operation performed by the user can be presented to the user.
- In one embodiment, for example, the target device may be capable of receiving the first operation instruction from the second input device. Thus, the operation of the target device can be controlled using the multiple input devices.
- In one embodiment, for example, the second input device may include at least one of a speech input device, a remote controller, or a mobile phone.
- In one embodiment, for example, a sound stating that the second input device may the speech input device, the remote controller, or the mobile phone may be outputted toward the user. Thus, the user can recognize a second input device suitable for the operation through the sound.[0024] In one embodiment, for example, the target device may be a television set, and the second input device includes a remote controller, and the recommendation for the second input device may be provided to the user by displaying on a display device of the television set an image i) indicating an appearance of the remote controller and ii) highlighting an operation portion capable of performing the first operation instruction, of the remote controller. Thus, the user can understand which part of the remote control serving as a second input device he or she should operate.
- In one embodiment, for example, the method may further include storing time information indicating a time when each of operation instructions has been received; and calculating a receiving frequency of the same type of operation instruction as the first operation instruction in a predetermined period of time, on the basis of the time information, to present to the user the recommendation for the second input device on the basis of the calculated receiving frequency. Thus, if the user performs the same operation multiple times within the predetermined period of time, a second input device suitable for the operation can be presented to the user.
- In one embodiment, for example, if the calculated receiving frequency is greater than or equal to a predetermined value, the second input device is provided to the user. Thus, if the user performs the same operation a predetermined number of times or more within the predetermined period of time, a second input device suitable for the operation can be presented to the user.
- In one embodiment, for example, the method may further include recognizing whether the first input device is the speech input device; recognizing whether the first instruction does not include an operating range for an operation indicated by the first operation instruction; when the first input device is recognized to be the speech input device and when the first instruction does not include the operating range, determining that the speech input device is not suitable to the type of the first operation instruction. By presenting a second input device different from the first input device to the user when there is no history information of the operation corresponding to the type of the first operation instruction, the user can avoid performing troublesome operations on the first input device.
- An electric device according to one embodiment of the present disclosure is an electric device, operation of the electric device capable to be controlled using a plurality of input devices The electric device includes a processor; and a non-transitory memory storing thereon a program, which executed by the processor, causes the processor to: receive, from one of the plurality of the input devices, a first operation instruction issued to the electric device, with a first data format; recognize the received first operation instruction and the first data format; determine that the one of the plurality of the input devices is a first input device corresponding to the recognized first data format; and provide to a user of the electric device a recommendation for a second input device, a type of the second input device being different from a type of the first input device, when it is determined that a type of the first operation instruction is identical to a type of a second operation instruction that is received from the second input device earlier than the reception of the first operation instruction from the first input device.
- Since a second input device different from the first input device is presented to the user when the user performs an operation which is not suitable for the first input device, the user can more easily perform the desired operation on the target device.
- A computer program according to one embodiment of the present disclosure is a non-transitory computer-readable recording medium storing a program for controlling an electric device, operation of the electric device capable to be controlled using a plurality of input devices. The computer program causing a computer of the electric device to: receive, from one of the plurality of the input devices, a first operation instruction issued to the electric device, with a first data format; recognize the received first operation instruction and the first data format; determine that the one of the plurality of the input devices is a first input device corresponding to the recognized first data format; and provide to a user of the electric device a recommendation for a second input device, a type of the second input device being different from a type of the first input device, when it is determined that a type of the first operation instruction is identical to a type of a second operation instruction that is received from the second input device earlier than the reception of the first operation instruction from the first input device. Since a second input device different from the first input device is presented to the user when the user performs an operation which is not suitable for the first input device, the user can more easily perform the desired operation on the target device.
- A method according to one embodiment of the present disclosure is a method for controlling an operation of a target device using a plurality of input devices including a speech input device. The method includes acquiring, from the speech input device, speech information including I) environmental sound around the speech input device and ii) a speech instruction indicating an operation instruction issued to the target device; calculating a level of noise included in the speech information; recognizing the operation instruction indicated by the speech instruction; recognizing a type of the operation instruction based on the recognition result of the operation instruction; and providing to a user of the target device a recommendation for a second input device on the basis of the calculated noise level and the recognized type of the operation instruction, wherein a type of the second input device does not include speech type. Since a second input device different from the speech input device is presented to the user on the basis of the noise level, it is possible to avoid performing a different operation from the operation intended by the user.
- In one embodiment, for example, the recommendation for the second input device may be determined based on i) the calculated noise level, ii) the recognized type of the operation instruction, and iii) recommended input device information, wherein the recommended input device information indicates an input device suitable to each operation instruction type. Thus, a second input device suitable for the environment in which noise is occurring can be presented to the user.
- In one embodiment, for example, if the noise level is higher than or equal to a predetermined value, the recommendation for the second input device may be provided to the user. By presenting a second input device different from the speech input device to the user when the noise level is higher than the predetermined value, it is possible to avoid performing a different operation from the operation intended by the user.
- A method according to one embodiment of the present disclosure is a method for controlling an operation of a target device using a plurality of input devices including a speech input device The method includes acquiring, from the speech input device, speech information including an operation instruction issued to the target device; recognizing the operation instruction included in the speech information; recognizing a type of the operation instruction based on the recognition result of the operation instruction; calculating a likelihood of the recognized operation instruction; and providing to a user of the target device a recommendation for a second input device on the basis of the recognized likelihood and the recognized type of the operation instruction, wherein a type of the second input device does not include speech type.
- Since a second input device different from the speech input device is presented to the user on the basis of the likelihood, it is possible to avoid performing a different operation from the operation intended by the user.
- In one embodiment, for example, the recommendation for the second input device may be determined based on i) the recognized likelihood, ii) the recognized type of the operation instruction, and iii) recommended input device information, wherein the recommended input device information indicates an input device suitable to each operation instruction type. Thus, a second input device suitable for a condition under which the likelihood of speech input is low can be presented to the user.
- In one embodiment, for example, the recommendation for the second input device may be determined based on i) the recognized likelihood, ii) the recognized type of the operation instruction, and iii) recommended input device information, wherein the recommended input device information indicates an input device suitable to each operation instruction type. By presenting a second input device different from the speech input device to the user when the likelihood is lower than the predetermined value, it is possible to avoid performing a different operation from the operation intended by the user.
- Now, an embodiment will be described with reference to the accompanying drawings. However, the embodiment described below is only illustrative. The numbers, shapes, elements, steps, the order of the steps, and the like described in the embodiment are also only illustrative and do not limit the technology of the present disclosure. Of the elements of the embodiment, elements which are not described in the independent claims representing the highest concept will be described as optional elements.
- First, there will be described an overview of services provided by an information management system according to the present embodiment.
-
FIG. 25A is a diagram showing an overview of the services provided by the information management system according to the present embodiment. The information management system includes agroup 600, a datacenter operating company 610, and aservice provider 620. - The
group 600 is, for example, a corporation, organization, or household and may have any size. Thegroup 600 includesmultiple devices 601 including first and second devices, and ahome gateway 602. Thedevices 601 include devices which can be connected to the Internet (e.g., smartphone, personal computer (PC), and television) and devices which cannot be connected to the Internet by themselves (e.g., lighting system, washer, and refrigerator). Thedevices 601 may include devices which cannot be connected to the Internet by themselves but can be connected thereto through thehome gateway 602. Users 6 use thedevices 601 in thegroup 600. - The data
center operating company 610 includes acloud server 611. Thecloud server 611 is a virtual server that cooperates with various devices through the Internet. Thecloud server 611 mostly manages big data or the like, which is difficult to handle using a typical database management tool or the like. The datacenter operating company 610 performs the management of the data, the management of thecloud server 611, the operation of a data center which performs those, and the like. Details of the operation performed by the datacenter operating company 610 will be described later. - The data
center operating company 610 is not limited to a corporation which only manages the data orcloud server 611. For example, as shown inFIG. 25B , if a device manufacturer which develops or manufactures one of thedevices 601 is also performing the management of the data orcloud server 611, or other operations, the device manufacturer serves as the datacenter operating company 610. Further, there may be multiple datacenter operating companies 610. For example, as shown inFIG. 25C , if a device manufacturer and a management company jointly manage the data orcloud server 611 or share such management, both or either one thereof serve as the datacenter operating company 610. - The
service provider 620 includes aserver 621. Theserver 621 may have any size and may be, for example, a memory in a personal computer (PC). Theservice provider 620 need not necessarily include theserver 621. - Further, the information management system need not necessarily include the
home gateway 602. For example, if thecloud server 611 manages all the data, the information management system does not have to include thehome gateway 602. There are also cases in which any device which cannot be connected to the Internet by itself does not exist, like cases in which all devices in the household are connected to the Internet. - Next, the flow of information in the information management system will be described.
- First, the first and second devices in the
group 600 transmit log information thereof to thecloud server 611 of the datacenter operating company 610. Thecloud server 611 accumulates the log information of the first and second devices (anarrow 631 inFIG. 25A ). As used herein, the log information is information indicating the operating state, operation date/time, or the like of each of thedevices 601. For example, The log information is the viewing history of a television, information about the programmed recording of a recorder, the operation date/time of a washer, the amount of laundry, the date/time or frequency of opening/closure of a refrigerator, but is not limited thereto. It may be various types of information available from various types of devices. The log information may be directly provided to thecloud server 611 by thedevices 601 through the Internet. The log information from thedevices 601 may be temporarily accumulated in thehome gateway 602 and then provided to thecloud server 611 by thehome gateway 602. - Subsequently, the
cloud server 611 of the datacenter operating company 610 provides a predetermined amount of the accumulated log information to theservice provider 620. The predetermined amount may be an amount obtained by compiling the information accumulated in the datacenter operating company 610 so that the information can be provided to theservice provider 620, or may be an amount requested by theservice provider 620. Further, the log information need not necessarily be provided in the predetermined amount, and the amount of the log information to be provided may be changed according to the situation. The log information is stored in theserver 621 held by theservice provider 620 as necessary (anarrow 632 inFIG. 25A ). - The
service provider 620 organizes the log information into information suitable for services to be provided to users and then provides the resulting information to the users. The users to which such information is provided may be the users 6, who use thedevices 601, or may beexternal users 7. The method for providing the information to theusers 6 or 7 may be, for example, to provide the information directly to theusers 6 or 7 by the service provider 620 (arrows FIG. 25A ). The method for providing information to the users 6 may also be to provide the information to the users 6 again through thecloud server 611 of the data center operating company 610 (arrows FIG. 25A ). Further, thecloud server 611 of the datacenter operating company 610 may organize the log information into information suitable for services to be provided to users and then provide the resulting information to theservice provider 620. - The users 6 may be the same as the
external users 7 or may differ therefrom. -
FIG. 1 is a drawing showing an overview of a speech operation system according to the present embodiment. - As shown in
FIG. 1 , the speech operation system according to the present embodiment includes a speech input device 1, aserver 2,target devices 3,input devices 4, and anetwork 5. The speech input device 1 acquires a speech generated by the user and transmits the acquired speech to theserver 2. Theserver 2 transmits or receives data to or from the speech input device 1,target devices 3, andinput devices 4. Eachtarget device 3 performs an operation indicated by a device operation command received from theserver 2 and determines whether there is any recommended input device suitable for that operation except for the speech input device. Eachinput device 4 receives input information from the user and transmits the position information of the input device. Thenetwork 5 connects the speech input device 1,server 2,target devices 3, andinput devices 4 to each other using a communication method such as a wired or wireless LAN or Bluetooth®. - The
target devices 3 are electric devices whose operation can be controlled using multiple input devices including the speech input device 1. Thetarget devices 3 include, for example, a television, a recorder, an air-conditioner, a lighting system, an audio system, a telephone, an intercommunication system, and the like. - In the present embodiment, one or both of the speech input device 1 and
server 2 may be incorporated into eachtarget device 3. -
FIG. 2 is a block diagram showing an example of the specific elements of the speech input device 1 according to the present embodiment. As shown inFIG. 2 , the speech input device 1 includes aspeech acquisition unit 101, aspeech detection unit 102, a speechsection cutout unit 103, a featurevalue calculation unit 104, and acommunication unit 105. Thespeech acquisition unit 101 detects a speech. Thespeech detection unit 102 determines whether the speech has been detected. The speechsection cutout unit 103 detects the section in which the speech inputted by the user exists, of the acquired speech signal and cuts out the detected speech section, since the speech signal includes noise or the like. The featurevalue calculation unit 104 calculates the feature value of the speech on the basis of the cut-out speech section. Thecommunication unit 105 transmits the calculated speech feature value to theserver 2. - The speech input device 1 is, for example, a microphone incorporated in or connected to each
target device 3, a microphone incorporated in a remote control included with eachtarget device 3 or the like, a microphone incorporated in or connected to a mobile communication terminal, or a sound concentrating microphone placed in the house. - At least some of the elements of the speech input device 1 can be implemented by a microcomputer and a memory. For example, the
speech detection unit 102, speechsection cutout unit 103, and featurevalue calculation unit 104 can be implemented by a microcomputer and a memory. In this case, the microcomputer performs the above processes on the basis of a computer program read from the memory. -
FIG. 3 is a block diagram showing an example of the specific elements of theserver 2 according to the present embodiment. As shown inFIG. 3 , theserver 2 includes acommunication unit 201, aspeech recognition unit 202, a speech recognitiondictionary storage unit 203, a deviceoperation determination unit 204, and a device operation determination table 205. Thecommunication unit 201 receives the speech feature value transmitted by thecommunication unit 105 of the speech input device 1 and transmits an operation command and the ID of the input device to thecorresponding target device 3. Thespeech recognition unit 202 converts the received speech feature value into character strings or word strings on the basis of information in the speech recognitiondictionary storage unit 203. The speech recognitiondictionary storage unit 203 are storing spoken sentences which may be inputted to thetarget devices 3, as shown inFIG. 8 . For example, it is storing device operation-related spoken sentences such as “power on” and “increase the volume” and small talk-related spoken sentences such as “good morning” and “tell the weather.” Details of the table shown inFIG. 8 will be described later. - The device
operation determination unit 204 determines an operation command from the result of the speech recognition on the basis of the device operation determination table 205. As shown inFIGS. 9A and 9B , the device operation determination table 205 is storing operation commands and input device IDs. For example, the device operation determination table 205 is storing the following information; the operation command for powering on the television is C001; the operation command for controlling the volume of the television is C002; the ID of the speech input device, to which a speech is inputted, is I001, and the ID of the remote control whose buttons are operated is I002. Details of the tables shown inFIG. 9A and 9B will be described later. - At least some of the elements of the
server 2 can be implemented by a microcomputer and a memory. For example, thespeech recognition unit 202, speech recognitiondictionary storage unit 203, deviceoperation determination unit 204, and device operation determination table 205 can be implemented by a microcomputer and a memory. In this case, the microcomputer performs the above processes on the basis of a computer program read from the memory. -
FIG. 4 is a block diagram showing an example of the specific elements of one of thetarget devices 3 according to the present embodiment. As shown in FIG. 4, thetarget device 3 includes acommunication unit 301, adevice control unit 302, a consecutiveoperation determination unit 303, an operation command storage table 304, a recommended inputdevice determination unit 305, a recommended input device determination table 306, a recommended input deviceposition determination unit 307, and an operationmethod display unit 308. - The
communication unit 301 receives the operation command and input device ID transmitted by theserver 2. Thedevice control unit 302 recognizes an operation instruction indicated by the received operation command, as well as recognizes an input device which has received an operation related to the operation instruction from the user, on the basis of the received input device ID. Thedevice control unit 302 then controls the operation of thetarget device 3 in accordance with the received operation command. The consecutiveoperation determination unit 303 determines whether the user has consecutively inputted operations, on the basis of the operation command storage table 304. As shown inFIG. 11A , the operation command storage table 304 is storing histories of operation commands inputted by the user. Details of the history information shown inFIG. 11A will be described later. - The recommended input
device determination unit 305 determines a recommended input device on the basis of the recommended input device determination table 306. As shown inFIGS. 13A and 13B , the recommended input device determination table 306 is storing operation commands and corresponding recommended input devices. For example, it is storing an operation command C001 and a corresponding recommended input device ID “UI001,” which represents the speech input device using speech input. Details of the tables shown inFIGS. 13A and 13B will be described later. - The recommended input device
position determination unit 307 determines the position of the recommended input device on the basis of information in a positioninformation acquisition unit 401 and then determines whether to present the user with the position of the recommended input device. The operationmethod display unit 308 presents the user with the recommended input device and the position thereof on the basis of the determinations made by the recommended inputdevice determination unit 305 and recommended input deviceposition determination unit 307. - At least some of the elements of the
target device 3 can be implemented by a microcomputer and a memory. For example, thedevice control unit 302, consecutiveoperation determination unit 303, operation command storage table 304, recommended inputdevice determination unit 305, recommended input device determination table 306, and recommended input deviceposition determination unit 307 can be implemented by a microcomputer and a memory. In this case, the microcomputer performs the above processes on the basis of a computer program read from the memory. -
FIG. 5 is a block diagram showing an example of the specific elements of one of theinput devices 4 according to the present embodiment. As shown inFIG. 5 , theinput device 4 includes the positioninformation acquisition unit 401, aninput unit 402, and acommunication unit 403. The positioninformation acquisition unit 401 acquires the position of theinput device 4 with respect to the user. Theinput unit 402 acquires information that the user has inputted by operating the buttons or by touching on the smartphone screen. Thecommunication unit 403 transmits the information acquired by the positioninformation acquisition unit 401 andinput unit 402 to theserver 2 orcorresponding target device 3. Theinput device 4 is, for example, one which receives input through the buttons, one which receives input through the touchscreen, or one which receives input through a gesture, but not limited thereto. It may be an input device of any type other than speech input type, as long as the input device can transmit an operation command to thetarget device 3. - At least some of the elements of the
input device 4 can be implemented by a microcomputer and a memory. For example, the positioninformation acquisition unit 401 andinput unit 402 can be implemented by a microcomputer and a memory. In this case, the microcomputer performs the above processes on the basis of a computer program read from the memory. -
FIG. 6 is a diagram showing a sequence in the speech operation system according to the present embodiment. - First, in step S001, the speech input device 1 calculates the feature value of a speech and transmits it to the
server 2. In step S002, theserver 2 performs a speech recognition process, that is, it converts the received speech feature value into character strings or word strings on the basis of the information in the speech recognition dictionary of the speech recognitiondictionary storage unit 203. In step S003, theserver 2 determines the type of a target device and a device operation intended by the user and transmits a corresponding operation command and input device ID to acorresponding target device 3. Details of the process in step S003 will be described later. - In step S004, the
target device 3 determines whether it can actually perform the operation command transmitted in step S003. For example, if thetarget device 3, which is a television, receives an operation command for controlling the volume with the television powered off, it cannot perform the operation. - In step S005, the
target device 3 determines whether the operation performed in step S004 is the same as the immediately preceding operation. Details of the process in step S005 will be described later. If thetarget device 3 determines in step S005 that operations have been performed consecutively, it performs a process in step S006. - In step S006, the
target device 3 determines whether, with respect to the operation instruction inputted by the user, there are anyother input devices 4 which are recommended over the speech input device 1. Details of the process in step S006 will be described later. If there are anyinput devices 4 which are recommended over the speech input device 1, therespective input devices 4 perform a process in step S007. - In step S007, the recommended
input devices 4 determined in step S006 each determine the position thereof. Examples of the method for determining the position include the following: if eachinput device 4 includes a pressure sensor, it determines whether the user is holding theinput device 4, based on whether there is a pressure; if eachinput device 4 includes a position sensor such as a GPS, RFID, or infrared ID, it determines the position using the position sensor; eachinput device 4 determines the position using the transmission/reception information of communication radio waves in a wireless LAN, Bluetooth, or the like; if thetarget device 3 includes a position acquisition unit equivalent to aninput device 4, the relative positions of theinput devices 4 andtarget device 3 are determined; and if thetarget device 3 includes a camera, the position of the user is estimated using camera information, and the relative positions of theinput devices 4, user, andtarget device 3 are determined. - Note that appropriate positions may be selected as the relative positions of the user,
input devices 4, andtarget device 3 in accordance with the device used or operation command. In step S008, a recommended input device which is most suitable for the operation command is shown to the user on the basis of the position information of the recommended input devices determined in step S007. Details of the process in step S008 will be described later. -
FIG. 7 is a flowchart showing the steps of the device operation determination process according to the present embodiment. In this process, theserver 2 determines whether the speech inputted by the user includes an operation instruction issued to atarget device 3, on the basis of the result of the speech recognition. - First, in step S301, the
server 2 receives the character stings or word strings, which are the result of the speech recognition. In step S302, theserver 2 determines whether the result of the speech recognition includes an operation to be performed on atarget device 3. Specifically, theserver 2 determines whether the result of the speech recognition includes the type of atarget device 3 such as a “television” or “air conditioner” or a phrase indicating an operation to be performed on atarget device 3, such as “”power on” or “increase the volume.” If so determined, the process proceeds to step S303; not so determined, the process proceeds to step S305. - In step S303, the
server 2 calls an operation command corresponding to the result of the speech recognition on the basis of the device operation determination table inFIG. 8 . In step S304, theserver 2 transmits, to thetarget device 3, the operation command called in step S303, as well as an input device ID recognized based on the input device ID table ofFIG. 9B and representing the speech input device. - In step S305, the speech operation system calls a function other than the target device operation function on the basis of the result of the speech recognition and then performs the called function. For example, if a speech “tell the weather” has been inputted by the user, the speech operation system calls a Q&A function rather than the device operation function. For example, in response to the speech “tell the weather” inputted by the user, the speech operation system makes a response “what area's weather do you want to know?” or a response “today's weather is sunny.”
-
FIG. 8 is a diagram showing the device operation determination table for determining a device operation command from the result of speech recognition according to the present embodiment. - In the device operation determination table, respective speech recognition results and corresponding operation commands are stored in the transverse (row) direction. Note that a device operation determination table may be generated for each
target device 3. These operation commands represent the types of operations inputted by the user. - For example, if a speech “power on” is inputted to the speech input device 1, an operation command “C001” is called from the device operation determination table of
FIG. 8 . Note that in the device operation determination table, multiple speech recognition results may be associated with a single operation command. This is because the user may generate a speech “power on” or may generate a speech “turn on the power” in inputting a speech to issue a command for powering on a target device, that is, even when the user intends the same operation, he or she may use a different speech expression. -
FIG. 9A is a diagram showing an operation command table, andFIG. 9B is a diagram showing an input device ID table. - In the operation command table shown in
FIG. 9A , respective operation commands and corresponding target device types and device operations are stored in the transverse (row) direction. In the input device ID table shown inFIG. 9B , respective input device IDs and corresponding input devices and input methods are stored in the transverse (row) direction. - For example, when a speech “power on” is inputted to the speech input device 1, an operation command C001, as well as an input device ID I001 are called. Note that there may be operation commands which include fine adjustments such as “increase the volume by 1” and “decrease the volume by 1.”
-
FIG. 10 is a flowchart showing the steps of the consecutive operation determination process according to the present embodiment. In this process, the consecutiveoperation determination unit 303 of thetarget device 3 determines whether the user has performed the same operations consecutively. - In step S501, the consecutive
operation determination unit 303 stores the operation command transmitted in step S304, in the operation number N002 of the operation command storage table shown inFIG. 11A . In step S502, the continuousoperation determination unit 303 determines whether any operation command is stored in the operation number N001 of the operation command storage table. If so determined, the process proceeds to step S503; if not so determined, the process proceeds to step S506. - If the respective operation commands stored in the operation numbers N001 and N002 are matched in step S503, the process proceeds to step S504. If the operation commands stored in the operation numbers N001 and N002 are not matched, the process proceeds to step S506.
- In step S504, the consecutive
operation determination unit 303 confirms that the operations are consecutive and then discards the operation command information stored in the operation numbers N001 and N002. The process proceeds to step S505. In step S505, the recommended inputdevice determination unit 305 performs a recommended input device presentation determination process, ending the consecutive operation determination process. - If the consecutive
operation determination unit 303 determines in step S502 or S503 that the operations are not consecutive, it performs step S506. It stores the operation command stored in the operation number N002, in the operation number N001. - Owing to the consecutive operation determination process, only when the user has performed operations consecutively using an input device other than a recommended input device set for each operation command, the recommended input device is presented to the user. Thus, for example, if the user who is viewing the television performs an operation “increase the volume by one stage,” to which an input device other than the speech input device is set as a recommended input device, using the speech input device only once, a message such as “use of the remote control is recommended in order to control the volume” does not appear on the screen. Accordingly, such a message is prevented from hindering the viewing of the television by the user. That is, such a message is prevented from appearing each time the user performs an operation using an input device other than the recommended input device, thereby reducing the annoyance of the user. Further, by presenting the recommended input device to the user, the user can learn selective use of an input device for each operation command.
- For example, if the user inputs a speech “power on” as the first operation, an operation command C001 is stored in the operation number N002 in step S501. Then, in step S502, it is determined that no operation command is stored in the operation number N001. Accordingly, the process proceeds to step S506 to store the operation command C001 stored in the operation number N002, in the operation number N001.
- Subsequently, if the user inputs a speech “increase the volume,” an operation command C002 is stored in the operation number N002 in step S501. In step S502, C001 is stored in the operation number N001. Accordingly, the process proceeds to step S503. In step S503, the operation numbers N001 and N002 are not matched, since C001 is stored in the operation number N001 and C002 is stored in the operation number N002. Accordingly, the process proceeds to step S506 to store C002 stored in the operation number N002, in the operation number N001.
- Subsequently, if the user inputs a speech “increase the volume” consecutively, the same operations as those when inputting the second speech are performed until step S503. In step S503, the operation numbers N001 and N002 are matched, since C002 is stored in the operation number N001 and C002 is stored in the operation number N002. Accordingly, the process proceeds to step S504. In step S504, C002 stored in both the operation numbers N001 and N002 is discarded. The process proceeds to S505 to perform a recommended input device presentation determination process. Thus, the consecutive operation determination process ends.
-
FIG. 11A is a diagram showing an example of the operation command storage table according to the present embodiment. The operation command storage table is storing history information indicating an operation command received earlier than the current operation command. Thus, the types of the operations inputted by the user can be recognized based on the stored operation commands. The operation command storage table is used in the consecutive operation determination process. - In the operation command storage table shown in
FIG. 11A , respective operation numbers and corresponding operation commands are stored in the transverse (row) direction. If the respective operation commands stored in the operation numbers N001 and N002 are matched in step S503, those operation commands are discarded. In contrast, if the operation commands stored in the operation numbers N001 and N002 are not matched in step S503, the operation command stored in the operation number N002 is stored in the operation number N001. - Note that the operation command storage table may include time information indicating the time when each operation command has been inputted.
FIG. 11B is a diagram showing an operation command storage table including such time information. If the operation command storage table includes time information, operation commands to be compared may be the latest operation command and an operation command which has been inputted within a predetermined period of time before the latest operation command has been inputted. Operation commands to be compared may also be the latest operation command and an operation command which has been inputted immediately before the latest operation command has been inputted. While the examples in which two operation numbers are stored are shown inFIGS. 11A and 11B , three or more operation numbers may be stored. -
FIG. 12 is a flowchart showing the steps of the recommended input device presentation determination process according to the present embodiment. In this process, when the user inputs a speech to the speech input device to operate the target device, the recommended inputdevice determination unit 305 determines whether any input device other than the speech input device is set as a recommended input device. - In step S601, the recommended input
device determination unit 305 receives the input device ID transmitted in step S304. In step S602, the recommended inputdevice determination unit 305 determines whether the input device ID is I001 representing the speech input device to determine whether the user has operated the target device using speech input. If so determined, the process proceeds to step S603; if not so determined, the process ends. In step S603, the recommended inputdevice determination unit 305 refers to information in the recommended input device determination table shown inFIG. 13A and recommended input device ID information shown inFIG. 13B to determine whether a recommended input device ID corresponding to the operation command of the device operation performed by the user represents the speech input device or any other input device. - If the recommended input device ID is UI001 representing the speech input device, the recommended input
device determination unit 305 can determine that it is appropriate that the user has inputted the speech to the speech input device to operate the device and that there is no need to recommend any other input device to the user. Accordingly, the recommended inputdevice determination unit 305 ends the recommended input device presentation determination process without performing a recommended input device position determination process. - In contrast, if the recommended input device ID is not UI001 representing the speech input device, the process proceeds to step S604. In step S604, the recommended input
device determination unit 305 determines that with respect to the device operation performed by the user, there is an input device other than the speech input device, and the recommended input deviceposition determination unit 307 performs a recommended input device position determination process. - For example, if the user inputs a speech “increase the volume” corresponding to an operation command C002 twice consecutively, the recommended input
device determination unit 305 receives the input device ID I001 representing the speech input device in step S304 each time. In step S602, the received input device ID is I001. Accordingly, the process proceeds to step S603. In step S603, an operation using the remote control or touchscreen is recommended over an operation using a speech with respect to C002 on the basis of the recommended input device determination table shown inFIG. 13A . Accordingly, the process proceeds to step S604 to perform a recommended input device position determination process. - For example, assume that the user operates the television using multiple input devices such as the speech input device, remote control, and smartphone. While the user can more easily input a keyword by using speech input when searching for a program, the user can more easily operate the television by using a fast-feedback device such as the remote control when controlling the volume as desired by the user. However, for a user who cannot selectively use the multiple input devices, he or she inputs a speech “increase the volume” or a speech “decrease the volume” a number of times in an attempt to control the volume using speech input. Thus, the user feels annoyed about speech input. In view of the foregoing, the recommended input device presentation determination process is performed. Thus, if input devices other than the speech input device are previously set as recommended input devices, the recommended input devices are presented to the user. Thus, the user can know whether the inputted operation command is suitable for speech input. As a result, the user can learn selective use of operation commands suitable for speech input and other input devices and thus can selectively use the recommended multiple input devices and reduce the annoyance associated with use of the multiple input devices.
-
FIG. 13A is a diagram showing an example of the recommended input device determination table, andFIG. 13B is a diagram showing an example of the recommended input device ID table. - In the recommended input device determination table, respective operation commands and corresponding recommended input device IDs are stored in the transverse (row) direction. In the recommended input device ID table, respective recommended input device IDs and corresponding input devices and input methods are stored in the transverse (row) direction.
-
FIG. 14 is a flowchart showing the steps of the recommended input device presentation process according to the present embodiment. In this process, when the user inputs a speech to the speech input device to operate the target device, the operationmethod display unit 308 presents, to the user, a recommended input device closer to the user on the basis of the recommended input device and the position information of the recommended input device. - In step S801, the operation
method display unit 308 receives the recommended input device IDs and recommended input device position information transmitted in steps S006 and S007, respectively. If it has received no recommended input device IDs or recommended input device position information, the operationmethod display unit 308 repeats step S801 until it receives them. If the operationmethod display unit 308 determines in step S802 that a single recommended input device ID has been received in step S801, the process proceeds to step S804. If the operationmethod display unit 308 determines in step S802 that multiple recommended input device IDs have been received in step S801, the process proceeds to step S804. In step S803, the operationmethod display unit 308 determines a recommended input device which is closer to the user, of the multiple recommended input device IDs and decides the determined recommended input device as a recommended input device to be presented to the user. Note that all the multiple recommended input devices may be presented to the user without performing steps S802 and S803. - In step S804, the operation
method display unit 308 presents the recommended input device other than the speech input device to the user, for example, using a method shown inFIG. 16 or 17 . Details ofFIGS. 16 and 17 will be described later. - In step S805, the operation
method display unit 308 determines a method for presenting the position information of the recommended input device to the user on the basis of information as to whether to present the position of the recommended input device and the presentation method information corresponding to the distance between the user and recommended input device described in a table shown inFIG. 15A . If the distance between the user and recommended input device is shortest, the user may be holding the recommended input device, or the recommended input device may be within the reach of the user. Accordingly, the position information of the recommended input device is not presented. In contrast, if the distance between the user and recommended input device is long, the process proceeds to step S806. The operationmethod display unit 308 presents the position information of the recommended input device to the user, as shown inFIG. 18 , on the basis of the presentation method ID corresponding to the distance between the user and recommended input device shown inFIG. 15A and the presentation device and presentation method corresponding to the presentation method ID shown inFIG. 15B . The presentation device or presentation method is not limited to the remote control, television, speech, or screen display and can take various forms. - For example, if the remote control and smartphone are recommended with respect to the operation instruction inputted by the user, the operation
method display unit 308 receives the respective recommended input device IDs and the position information thereof in step S801. In step S802, the number of recommended input device IDs, which correspond to the remote control and smartphone, is two. Accordingly, the process proceeds to step S803. If the distance between the remote control and user is 50 cm and the distance between the smartphone and user is 80 cm, the operationmethod display unit 308 determines in step S803 that the remote control is closer to the user than the smartphone. In step S804, the operationmethod display unit 308 presents recommended input device information such as “remote control is recommended” to the user on the basis of the determination in step S803. In step S805, the remote control is distant from the user by 50 cm. Accordingly, the process proceeds to step S806. The operationmethod display unit 308 presents recommended input device position information to the user by causing the remote control to emit a sound. - If a recommended input device which is distant from the user is presented to the user, the user must take time and effort to move to search for or acquire the recommended input device. Thus, the user may feel annoyed about the use of the recommended input device. In view of the foregoing, if multiple recommended input devices are previously set, the operation
method display unit 308 presents, to the user, a recommended input device which is closest to the user, of the recommended input devices on the basis of the positions of the user and recommended input devices. Thus, even if all the recommended input devices are distant from the user, the user can know the position of the recommended input device which is the closest to the user. As a result, the user no longer has to take the time and effort to move to search for or acquire the recommended input device which is distant from the user. Further, presenting a recommended input device to the user can assist the user in selectively using the speech input device and the multiple input devices including the remote control. -
FIG. 15A is a diagram showing an example of a presentation method determination table, andFIG. 15B is a diagram showing an example of a presentation method ID table. - In the presentation method determination table, the respective distances D between the user and recommended input device and corresponding information as to whether to present the position information of the recommended input device and presentation method IDs are stored in the transverse (row) direction. Note that while the information as to whether to present the position information and the presentation method are changed at 0.3 m and 1.0 m in
FIG. 15A , these distances are not limiting. Further, while the range of the distance between the user and recommended input device are divided into three ranges and information as to whether to present the position information and the presentation method vary among the three ranges inFIG. 15A , there may be used other configurations, for example, one in which the range of the distance between the user and recommended input device is divided into two or four or more ranges and information as to whether to present the position information and the presentation method vary among such ranges. In the presentation method determination table, respective presentation method IDs and corresponding presentation devices and presentation methods are stored in the transverse (row) direction. The stored presentation devices and presentation methods are, for example, devices and methods shown inFIGS. 16, 17 and 18A to 18C . -
FIG. 16 is a drawing showing an example of a user interface (UI) for presenting a recommended input device to the user in the speech operation system according to the present embodiment. These drawings show examples of a UI which when there is a recommended input device that is more suitable for inputting an device operation than speech input, presents such a recommended input device to the user. In these examples, the user who has inputted speeches consecutively to change the volume of the television as thetarget device 3 is being recommended to operate the television using the remote control. As shown in Part (A) ofFIG. 16 , a display screen stating “remote control operation is recommended” may be displayed on the television. Or, as shown in Part (B) ofFIG. 16 , a sound stating “remote control operation is recommended” may be emitted from a speaker included in the television or remote control. Yet another presentation method may be to present the user with new information indicating that there is another recommended input device rather than presenting specific information indicating that “remote control operation is recommended” and then to present the user with detailed information only when the user needs it. -
FIG. 17 is a drawing showing an example of a method for specifically presenting the user with the operation method of a recommended input device in the speech operation system according to the present embodiment. These drawings show examples of a method for, when there is a recommended input device that is more suitable for inputting a device operation than speech input, specifically presenting the user with the operation method of the recommended input device. For example, the user who has inputted speeches consecutively to change the volume of the television is recommended to operate the buttons of the remote control. As shown in Part (A) ofFIG. 17 , the volume control buttons of the remote control may be allowed to light up. As shown in Part (B) ofFIG. 17 , another presentation method may be to display a remote control image on the television screen and to highlight portions corresponding to the volume control buttons in the displayed remote control image. Thus, the user can understand which buttons of the remote control should be pressed to perform the function which the user has performed using speech input. -
FIG. 18 is a drawing showing an example of a method for presenting the user with the position of a recommended input device in the speech operation system according to the present embodiment. These drawings show examples of a method for, when there is a recommended input device that is more suitable for inputting a device operation than speech input, presenting the user with the position of the recommended input device. For example, as shown in Part (A) ofFIG. 18 , the user may be informed of the position of the remote control by lighting the remote control itself. Alternatively, as shown in Part (B) ofFIG. 18 , the user may be informed of the position of the remote control by causing the remote control to emit a sound. Alternatively, as shown in Part (C) ofFIG. 18 , the position of the remote control may be presented by displaying a sketch of the room on the television screen. Note that instead of presenting the precise position of the remote control, the range in which the remote control is more likely to be placed, the direction in which the remote control is placed with respect to the television or user, or the like may be presented. - Hereafter, the above processes will be described using specific examples. For example, assume that the user has issued a speech “increase the volume” toward the television. First, the speech input device 1 performs the feature value extraction process S001 on the speech signal issued by the user. In the speech recognition process S002, the
server 2 converts the extracted speech feature value into character strings or word strings on the basis of the information in the speech recognitiondictionary storage unit 203. Theserver 2 then performs the device operation determination process S003 on the basis of the resulting character strings or word strings. Thus, theserver 2 determines that the speech issued by the user intends a “volume control” operation. Then, thetarget device 3 performs the device operation process S004 on the basis of the determination to increase the volume of the television. Thetarget device 3 then performs the consecutive operation determination process S005. Specifically, thetarget device 3 determines whether volume control operation commands have been performed consecutively, determines that the volume control operation command is the first one, and stores the operation command in the operation command storage table 304, ending the process. - Subsequently, for example, the user feels that the single volume control operation is not sufficient and then issues another speech “increase the volume” toward the television. Then, the speech input device 1,
server 2, andtarget device 3 perform similar processes to those which have been performed on the first speech until the consecutive operation determination process S005. Since the volume control operation command is stored in the operation number N001 of the operation command storage table 304, thetarget device 3 determines that the user has inputted the two volume control operation commands consecutively. That is, it determines that the user has inputted the same operation commands consecutively. Accordingly, thetarget device 3 performs the recommended input device presentation determination process S006. Specifically, it determines whether there are any recommended input devices other than the speech input device in controlling the volume. Thetarget device 3 then determines that the remote control or smartphone is more suitable for controlling the volume than speech input, and theinput devices 4 perform the recommended input device position determination process S007. As a result, theinput devices 4 calculate the distance between the user and remote control, for example, as 25 cm and the distance between the user and smartphone, for example, as 50 cm. Thus, thetarget device 3 determines that the recommended input device to be presented to the user is the remote control. Thetarget device 3 then performs the recommended input device presentation process S008 on the basis of the determination. Specifically, it causes the television to emit a sound stating “remote control operation is recommended” toward the user. - By presenting the recommended input device in this manner, it is possible to present a fast-feedback input device such as the remote control to the user with respect to a troublesome operation such as volume control, in which the user must input an speech “increase the volume” or a speech “decrease the volume” to the speech input device a number of times. Thus, the user can learn that the remote control or smartphone is recommended in controlling the volume. Such learning finally allows the user to select a recommended input device for each operation by himself or herself.
- In the above example, when the user has inputted the same operation instructions consecutively using speeches, the
target device 3 performs the recommended input device presentation determination process. Next, there will be described a process of performing the recommended input device presentation determination process on the basis of the frequency with which the same operation command has been inputted. -
FIG. 19 is a flowchart showing the steps of an input frequency determination process. In this process, thetarget device 3 calculates the frequency with which the same type of operation command as the current operation command has been inputted within a predetermined period of time and, if the calculated input frequency is greater than or equal to a predetermined value, presents a recommended input device to the user. - In step S901, the
target device 3 stores the operation command transmitted in step S304 in an operation command storage table (history information) including time information as shown inFIG. 11B . - In step S902, the consecutive
operation determination unit 303 calculates the frequency with which the same type of operation command as the current operation command has been inputted within the predetermined period of time (input frequency). The predetermined period of time may be any period of time and is set to, for example, 10 sec, 1 min, 30 min, or 1 h in accordance with the type of the target device. In the example ofFIG. 11B , the same type of two operation commands have been inputted within one min. - In step S903, the consecutive
operation determination unit 303 determines whether the input frequency is greater than or equal to a predetermined value. The predetermined value may be any value and is set to, for example, two. If the input frequency is greater than or equal to the predetermined value, the recommended inputdevice determination unit 305 performs the recommended input device presentation determination process in step S904, and the operationmethod display unit 308 presents a recommended input device to the user on the basis of the determination. For example, the operationmethod display unit 308 presents the remote control or smartphone, which is an input device other than the speech input device, to the user as a recommended input device. If the input frequency is not greater than or equal to the predetermined value, the recommended inputdevice determination unit 305 ends the process without performing the recommended input device presentation determination process. - As seen above, if the user has controlled the volume of the television using speech input a number of times within a short period of time, it is possible to present a fast-feedback input device such as the remote control to the user with respect to the troublesome operation using speech input. Thus, the user can learn that the remote control or smartphone is recommended in controlling the volume. Such learning finally allows the user to select a recommended input device for each operation by himself or herself.
- Next, there will be described a process of learning the operation of the target device which has changed in accordance with an operation command and controlling the target device on the basis of the learned operation.
-
FIG. 20 is a flowchart showing the steps of a learning process. In this process, thetarget device 3 manages, as history information, the operation thereof which has changed in accordance with an operation command. - In step S1001, the
communication unit 301 of thetarget device 3 receives an operation command. In step S1002, thedevice control unit 302 controls the operation of thetarget device 3. For example, if the user inputs a speech stating “increase the volume by 10” when the volume of the television is 15, thedevice control unit 302 increases the volume of the television by 10, thereby setting the volume to 25. In step S1003, thedevice control unit 302 stores, as history information, the operation in which the television volume of 15 has been changed to 25. The operation may be stored in the operation command storage table 304 serving as history information or may be stored in history information which is different from the operation command storage table 304. For another example, if the user inputs a speech stating “change the volume to 25,” thedevice control unit 302 sets the volume of the television to 25. Thedevice control unit 302 then stores, as history information, the operation in which the volume of the television has been changed to 25. By accumulating operations in this manner, it is possible to obtain a learning result indicating that “when increasing the volume, the volume is often changed to 25.” By controlling the device on the basis of such a learning result, it is possible to set the device in accordance with the preferences of the user. Further, since the history information includes specific numerals such as the amount of variation of volume, it is possible to accurately set the device in accordance with the preferences of the user. -
FIG. 21 is a flowchart showing the steps of a process of determining operations corresponding to an operation command. - In step S1101, the
communication unit 301 receives an operation command. In step S1102, thedevice control unit 302 determines whether it can operate the device in accordance with the operation command, which is based on speech input. For example, if a speech stating “increase the volume by 10” has been inputted, thedevice control unit 302 can operate the device, since the amount of volume to be increased is obvious. Accordingly, in step S1105, thedevice control unit 302 increases the volume of the television by 10. - On the other hand, if a speech stating “increase the volume” has been inputted, the
device control unit 302 determines that it cannot operate the device, since the amount of volume to be increased is not obvious. Accordingly, the process proceeds to step S1103. - In step S1103, the
device control unit 302 determines whether history information of operations corresponding to commands indicating “increase the volume” is accumulated. For example, thedevice control unit 302 determines whether corresponding multiple operations are accumulated. If so determined, thedevice control unit 302 operates the device in accordance with the operations in step S1105. For example, if operations in which the volume has been changed to 25 are accumulated, thedevice control unit 302 sets the volume of the television to 25. - If not so determined, the recommended input
device determination unit 305 performs a recommended input device presentation determination process in step S1104, and the operationmethod display unit 308 presents a recommended input device to the user on the basis of the determination. Thus, the user, who may control the volume using speech input a number of times, can avoid performing such troublesome operations. The recommended input device presented is, for example, the remote control or smartphone, which is an input device other than the speech input device. -
FIG. 22 is a flowchart showing another example of a process of determining an operation corresponding to an operation command. - In step S1201, the
communication unit 301 receives an operation command. In step S1202, thedevice control unit 302 determines whether the operation command, which is based on speech input, includes numerical information. The numerical information represents a specific numeral indicating the amount of variation by which the operation of the target device is changed. For example, if a speech stating “increase the volume by 10” has been inputted, the corresponding operation command includes the numeral of 10, by which the volume is to be increased. Accordingly, in step S1205, thedevice control unit 302 increases the volume of the television by 10. - In contrast, if the operation command does not include any numerical information, for example, if a speech stating “increase the volume” has been inputted, the process proceeds to step S1203. In step S1203, the
device control unit 302 determines whether numerical history information satisfying a condition is accumulated. For example, if there is a condition that the current volume of the television is 15 and if history information that the television volume of 15 has been changed to 25 by increasing it by 10 is accumulated, thedevice control unit 302 sets the volume of the television to 25 by increasing it by 10 in step S1205. - In contrast, if the
device control unit 302 determines that any history information satisfying the condition is not accumulated, the recommended inputdevice determination unit 305 performs a recommended input device presentation determination process in step S1204, and the operationmethod display unit 308 presents a recommended input device to the user on the basis of the determination. Thus, the user, who may control the volume using speech input a number of times, can avoid performing such troublesome operations. The recommended input device presented is, for example, the remote control or smartphone, which is an input device other than the speech input device. - Next, there will be described a process of performing a recommended input device presentation determination process on the basis of the likelihood of speech recognition.
FIG. 23 is a flowchart showing the steps of a speech recognition likelihood determination process. In this process, an operation instruction is recognized from speech information; the likelihood of the recognized operation instruction is calculated; if the likelihood is lower than a predetermined value, a recommended input device suitable for the corresponding operation command is presented to the user. - In step S1301, the
speech recognition unit 202 converts a received speech feature value into character strings or word strings. In step S1302, the deviceoperation determination unit 204 recognizes an operation command on the basis of the result of the speech recognition. Note that if thetarget device 3 has the functions of thespeech recognition unit 202 and deviceoperation determination unit 204, it performs these processes. - In step S1303, the
device control unit 302 calculates the likelihood of the speech recognition. For example, the likelihood can be obtained by calculating the distance between the recognized speech and a language model serving as a reference. In this case, the likelihood is higher as the calculated distance is shorter. - Subsequently, in step S1304, the
device control unit 302 determines whether the calculated likelihood is lower than a predetermined value. The predetermined value may be any value. If the likelihood is higher than or equal to the predetermined value, thedevice control unit 302 controls the device in accordance with the recognized operation command in step S1306. - In contrast, if the likelihood is lower than the predetermined value, the recommended input
device determination unit 305 performs a recommended input device presentation determination process in step S1305, and the operationmethod display unit 308 presents a recommended input device to the user on the basis of the determination. The recommended input device presented is, for example, the remote control or smartphone, which is an input device other than the speech input device. If the likelihood is low, an operation based on the inputted speech is more likely to be different from the operation intended by the user. In this case, by presenting a recommended input device, it is possible to avoid performing the operation different from the operation intended by the user. - Next, there will be described a process of performing a recommended input device presentation determination process on the basis of the noise level of an acquired speech.
FIG. 24 is a flowchart showing the steps of a noise level determination process. In this process, the noise level of speech information is calculated, and if the noise level is higher than or equal to a predetermined value, a recommended input device suitable for the corresponding operation command is presented to the user. - In step S1401, the
speech recognition unit 202 acquires speech information including a speech feature value. This speech information includes the environmental sound around the speech input device and an operation instruction issued to the target device. In step S1402, thespeech recognition unit 202 calculates the noise level of the speech information. For example, the noise level can be calculated by obtaining S/N by making comparison between the sound pressures of the noise and speech. In this case, the noise level is higher as S/N is lower. - Subsequently, in step S1403, the
speech recognition unit 202 determines whether the calculated noise level is higher than or equal to a predetermined value. The predetermined value may be any value. If the noise level is lower than the predetermined value, thedevice control unit 302 controls the target device in accordance with an operation command recognized from the speech information in step S1405. - In contrast, if the noise level is higher than or equal to the predetermined value, the recommended input
device determination unit 305 performs a recommended input device presentation determination process in step S1404, and the operationmethod display unit 308 presents a recommended input device to the user on the basis of the determination. The recommended input device presented is, for example, the remote control or smartphone, which is an input device other than the speech input device. If the noise level is high, an operation based on the inputted speech is more likely to be different from the operation intended by the user. In this case, by presenting a recommended input device, it is possible to avoid performing the operation different from the operation intended by the user. - Note that a recommended input device presentation determination process may be performed based on both the noise level and likelihood.
- The processes described in the present embodiment may be performed by any of hardware, software, and a combination thereof. A computer program for causing hardware or the like to perform those processes may be stored in a memory and then executed by a microcomputer. Such a computer program may be installed from a recording medium (semiconductor memory, optical disk, etc.) storing the computer program to the respective devices or may be downloaded through an electrical communication line such as the Internet. Such a computer program may also be wirelessly installed into the respective devices.
- For example, the technology described in the above aspect can be implemented in the following types of cloud services. However, these types of cloud services are only illustrative.
-
FIG. 26 is a diagram showing an overview of services provided by an information management system of service type 1 (own data center type cloud services). In the present type, aservice provider 620 acquires information from agroup 600 and provides services to a user. In the present type, theservice provider 620 has the functions of a data center operating company. That is, theservice provider 620 holds a cloud server which manage big data. Accordingly, there is no data center operating company. - In the present type, the
service provider 620 operates and manages a data center (cloud server) 703. Theservice provider 620 also manages an operating system (OS) 702 and anapplication 701. Theservice provider 620 provides services using theOS 702 and application 701 (arrow 704). - Service Type 2: laaS Cloud Services
-
FIG. 27 is a diagram showing an overview of services provided by an information management system of service type 2 (laaS cloud services). As used herein, laaS is the abbreviation of infrastructure as a service and is a cloud service providing model which provides an infrastructure for constructing and operating a computer system, as a service through the Internet. - In the present type, a data
center operating company 610 operates and manages a data center (cloud server) 703. Aservice provider 620 manages anOS 702 and anapplication 701. Theservice provider 620 provides services using theOS 702 and application 701 (arrow 704). -
FIG. 28 is a diagram showing an overview of services provided by an information management system of service type 3 (PaaS cloud services). As used herein, PaaS is the abbreviation of platform as a service and is a cloud service providing model which provides a platform serving as an infrastructure for constructing and operating software, as a service through the Internet. - In the present type, a data
center operating company 610 manages anOS 702 and operates and manages a data center (cloud server) 703. Aservice provider 620 manages anapplication 701. Theservice provider 620 provides services using theOS 702 and application 701 (arrow 704). -
FIG. 29 is a diagram showing an overview of services provided by an information management system of service type 4 (SaaS cloud services). As used herein, SaaS is the abbreviation of software as a service. SaaS cloud services are, for example, a cloud service providing model which allows users not holding a data center (cloud server) such as companies or individuals to use an application provided by a platform provider holding a data center (cloud server) through a network such as the Internet. - In the present type, a data
center operating company 610 manages anapplication 701 and anOS 702 and operates and manages a data center (cloud server) 703. Aservice provider 620 provides services using theOS 702 and application 701 (arrow 704). - As seen above, in any of the cloud services types, the
service provider 620 provides services. The service provider or data center operating company may develop an OS, application, a database of big data, or the like on its own or may outsource such development to a third party. - The technology of the present disclosure is particularly useful in the field of technologies that control the operation of a device using a speech.
Claims (3)
1. A method for controlling an operation of a target device using a plurality of input devices including a speech input device, the method comprising:
acquiring, from the speech input device, speech information including i) environmental sound around the speech input device, and ii) a speech instruction indicating an operation to be performed on the target device;
calculating a level of noise included in the speech information;
recognizing the operation; and
informing a user of a second input device as a recommended input device based on the calculated noise level and the recognized operation instruction,
wherein the second input device does not include a speech input device.
2. The method according to claim 1 ,
wherein the recommendation for the second input device is determined based on i) the calculated noise level, ii) the recognized operation, and iii) recommended input device information, and
wherein the recommended input device information indicates an input device suitable to each operation to be performed on the target device.
3. The method according to claim 1 ,
wherein if the noise level is higher than or equal to a predetermined value, the user is informed of the second input device as the recommended input device.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/622,126 US20170289582A1 (en) | 2014-07-01 | 2017-06-14 | Device control method and electric device |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2014-135899 | 2014-07-01 | ||
JP2014135899 | 2014-07-01 | ||
JP2015-050967 | 2015-03-13 | ||
JP2015050967A JP6482911B2 (en) | 2014-07-01 | 2015-03-13 | Device control method and electric device |
US14/743,704 US9721572B2 (en) | 2014-07-01 | 2015-06-18 | Device control method and electric device |
US15/622,126 US20170289582A1 (en) | 2014-07-01 | 2017-06-14 | Device control method and electric device |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/743,704 Continuation US9721572B2 (en) | 2014-07-01 | 2015-06-18 | Device control method and electric device |
Publications (1)
Publication Number | Publication Date |
---|---|
US20170289582A1 true US20170289582A1 (en) | 2017-10-05 |
Family
ID=53496450
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/743,704 Expired - Fee Related US9721572B2 (en) | 2014-07-01 | 2015-06-18 | Device control method and electric device |
US15/622,126 Abandoned US20170289582A1 (en) | 2014-07-01 | 2017-06-14 | Device control method and electric device |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/743,704 Expired - Fee Related US9721572B2 (en) | 2014-07-01 | 2015-06-18 | Device control method and electric device |
Country Status (3)
Country | Link |
---|---|
US (2) | US9721572B2 (en) |
EP (2) | EP3252730A1 (en) |
JP (1) | JP6482911B2 (en) |
Families Citing this family (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6482911B2 (en) * | 2014-07-01 | 2019-03-13 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカPanasonic Intellectual Property Corporation of America | Device control method and electric device |
EP3591648B1 (en) * | 2015-05-19 | 2022-07-06 | Sony Group Corporation | Information processing apparatus, information processing method, and program |
US9653075B1 (en) * | 2015-11-06 | 2017-05-16 | Google Inc. | Voice commands across devices |
US10796693B2 (en) * | 2015-12-09 | 2020-10-06 | Lenovo (Singapore) Pte. Ltd. | Modifying input based on determined characteristics |
US9911417B2 (en) * | 2016-04-01 | 2018-03-06 | Tai-An Lu | Internet of things system with voice-controlled functions and method for processing information of the same |
KR20170132622A (en) * | 2016-05-24 | 2017-12-04 | 삼성전자주식회사 | Electronic device having speech recognition function and operating method of Electronic device |
KR102505719B1 (en) * | 2016-08-12 | 2023-03-03 | 삼성전자주식회사 | Electronic device and method for recognizing voice of speech |
CN107765838A (en) * | 2016-08-18 | 2018-03-06 | 北京北信源软件股份有限公司 | Man-machine interaction householder method and device |
JP6659514B2 (en) | 2016-10-12 | 2020-03-04 | 東芝映像ソリューション株式会社 | Electronic device and control method thereof |
US10332523B2 (en) * | 2016-11-18 | 2019-06-25 | Google Llc | Virtual assistant identification of nearby computing devices |
US10643609B1 (en) * | 2017-03-29 | 2020-05-05 | Amazon Technologies, Inc. | Selecting speech inputs |
KR102025391B1 (en) * | 2017-05-15 | 2019-09-25 | 네이버 주식회사 | Device control according to user's talk position |
CN107146616B (en) * | 2017-06-13 | 2020-05-08 | Oppo广东移动通信有限公司 | Equipment control method and related product |
US11205421B2 (en) * | 2017-07-28 | 2021-12-21 | Cerence Operating Company | Selection system and method |
JP6997554B2 (en) * | 2017-08-03 | 2022-01-17 | シャープ株式会社 | Home appliance system |
JP6843723B2 (en) * | 2017-09-29 | 2021-03-17 | シャープ株式会社 | Server equipment, server client system and programs |
JP2019068319A (en) * | 2017-10-03 | 2019-04-25 | 東芝ライフスタイル株式会社 | Consumer-electronics system |
CN108130683A (en) * | 2017-12-21 | 2018-06-08 | 迪尔阿扣基金两合公司 | A kind of household electrical appliance and the method for running household electrical appliance |
JP6928842B2 (en) * | 2018-02-14 | 2021-09-01 | パナソニックIpマネジメント株式会社 | Control information acquisition system and control information acquisition method |
US10848392B2 (en) * | 2018-02-15 | 2020-11-24 | Lenovo (Singapore) Pte. Ltd. | Systems and methods to use digital assistant to join network |
US10685652B1 (en) * | 2018-03-22 | 2020-06-16 | Amazon Technologies, Inc. | Determining device groups |
JP2019198027A (en) * | 2018-05-10 | 2019-11-14 | パナソニックIpマネジメント株式会社 | Control system and program |
JP6700338B2 (en) * | 2018-06-19 | 2020-05-27 | ソフトバンク株式会社 | Information processing apparatus, information processing method, information processing program |
JP7224863B2 (en) * | 2018-11-09 | 2023-02-20 | キヤノン株式会社 | Relay server and control method |
JP2020099459A (en) * | 2018-12-21 | 2020-07-02 | パナソニックIpマネジメント株式会社 | Washing machine |
JP7145502B2 (en) * | 2018-12-28 | 2022-10-03 | パナソニックIpマネジメント株式会社 | CONTROLLER, LIGHTING SYSTEM, LIGHTING CONTROL METHOD AND PROGRAM |
JP2020112692A (en) * | 2019-01-11 | 2020-07-27 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカPanasonic Intellectual Property Corporation of America | Method, controller and program |
KR102253524B1 (en) * | 2019-09-02 | 2021-05-20 | 네이버 주식회사 | Method and system for applying loudness normalization |
TWI734340B (en) * | 2020-01-10 | 2021-07-21 | 諾利嘉股份有限公司 | Control device for smart electrical equipment |
CN112489413B (en) * | 2020-11-27 | 2022-01-11 | 京东方科技集团股份有限公司 | Control method and system of remote controller, storage medium and electronic equipment |
EP4210343A4 (en) * | 2020-12-04 | 2024-03-06 | Samsung Electronics Co., Ltd. | Control device and operation method thereof |
WO2024049210A1 (en) * | 2022-08-30 | 2024-03-07 | 엘지전자 주식회사 | Power management device, and power management method using same |
CN116304320A (en) * | 2023-03-06 | 2023-06-23 | 深圳绿米联创科技有限公司 | Spatial template recommendation method and device, electronic equipment and storage medium |
Citations (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6219645B1 (en) * | 1999-12-02 | 2001-04-17 | Lucent Technologies, Inc. | Enhanced automatic speech recognition using multiple directional microphones |
US20020043557A1 (en) * | 2000-07-05 | 2002-04-18 | Tetsuya Mizoguchi | Remote controller, mobile phone, electronic apparatus, and method of controlling the electrical apparatus |
US20030065515A1 (en) * | 2001-10-03 | 2003-04-03 | Toshikazu Yokota | Information processing system and method operable with voice input command |
US20030103088A1 (en) * | 2001-11-20 | 2003-06-05 | Universal Electronics Inc. | User interface for a remote control application |
US20030229900A1 (en) * | 2002-05-10 | 2003-12-11 | Richard Reisman | Method and apparatus for browsing using multiple coordinated device sets |
US20120316876A1 (en) * | 2011-06-10 | 2012-12-13 | Seokbok Jang | Display Device, Method for Thereof and Voice Recognition System |
US20130325451A1 (en) * | 2012-05-31 | 2013-12-05 | Elwha LLC, a limited liability company of the State of Delaware | Methods and systems for speech adaptation data |
US20140169795A1 (en) * | 2009-01-30 | 2014-06-19 | Altorr Corporation | Smartphone control of electrical devices |
US20150116096A1 (en) * | 2013-03-13 | 2015-04-30 | Panasonic Corporation | Information apparatus, apparatus to be controlled, radio communication system, and method for selecting apparatus to be controlled |
US20160005404A1 (en) * | 2014-07-01 | 2016-01-07 | Panasonic Intellectual Property Corporation Of America | Device control method and electric device |
US20160156508A1 (en) * | 2014-12-01 | 2016-06-02 | Gwangju Institute Of Science And Technology | Integrated network controlling system, network controlling apparatus and method, and a readable medium containing the method |
US20160259656A1 (en) * | 2015-03-08 | 2016-09-08 | Apple Inc. | Virtual assistant continuity |
US20170061784A1 (en) * | 2015-08-27 | 2017-03-02 | Altorr Corporation | Sip-and-puff Controller of Electrical Devices for Paraplegic or Quadriplegic Humans |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0451799A (en) * | 1990-06-20 | 1992-02-20 | Hitachi Ltd | Remote control system |
JPH07104962A (en) * | 1993-10-06 | 1995-04-21 | Toshiba Corp | Interactive system |
JP2001307540A (en) | 2000-04-26 | 2001-11-02 | Toshiba Lighting & Technology Corp | Lighting system |
US7574356B2 (en) | 2004-07-19 | 2009-08-11 | At&T Intellectual Property Ii, L.P. | System and method for spelling recognition using speech and non-speech input |
US20060235701A1 (en) | 2005-04-13 | 2006-10-19 | Cane David A | Activity-based control of a set of electronic devices |
JP2009271835A (en) * | 2008-05-09 | 2009-11-19 | Toyota Central R&D Labs Inc | Equipment operation controller and program |
JP2012132052A (en) | 2010-12-20 | 2012-07-12 | Panasonic Corp | Rare earth magnet recovery method |
US9514737B2 (en) * | 2011-09-13 | 2016-12-06 | Mitsubishi Electric Corporation | Navigation apparatus |
KR20130116107A (en) * | 2012-04-13 | 2013-10-23 | 삼성전자주식회사 | Apparatus and method for remote controlling terminal |
JP5949413B2 (en) | 2012-10-04 | 2016-07-06 | 日本電気株式会社 | User instruction recognition system, user instruction recognition method, information processing apparatus, portable terminal, control method thereof, and control program |
-
2015
- 2015-03-13 JP JP2015050967A patent/JP6482911B2/en not_active Expired - Fee Related
- 2015-06-18 US US14/743,704 patent/US9721572B2/en not_active Expired - Fee Related
- 2015-06-23 EP EP17183297.5A patent/EP3252730A1/en not_active Withdrawn
- 2015-06-23 EP EP15173226.0A patent/EP2963630B1/en not_active Not-in-force
-
2017
- 2017-06-14 US US15/622,126 patent/US20170289582A1/en not_active Abandoned
Patent Citations (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6219645B1 (en) * | 1999-12-02 | 2001-04-17 | Lucent Technologies, Inc. | Enhanced automatic speech recognition using multiple directional microphones |
US20020043557A1 (en) * | 2000-07-05 | 2002-04-18 | Tetsuya Mizoguchi | Remote controller, mobile phone, electronic apparatus, and method of controlling the electrical apparatus |
US20030065515A1 (en) * | 2001-10-03 | 2003-04-03 | Toshikazu Yokota | Information processing system and method operable with voice input command |
US20030103088A1 (en) * | 2001-11-20 | 2003-06-05 | Universal Electronics Inc. | User interface for a remote control application |
US20030229900A1 (en) * | 2002-05-10 | 2003-12-11 | Richard Reisman | Method and apparatus for browsing using multiple coordinated device sets |
US20140169795A1 (en) * | 2009-01-30 | 2014-06-19 | Altorr Corporation | Smartphone control of electrical devices |
US20120316876A1 (en) * | 2011-06-10 | 2012-12-13 | Seokbok Jang | Display Device, Method for Thereof and Voice Recognition System |
US20130325451A1 (en) * | 2012-05-31 | 2013-12-05 | Elwha LLC, a limited liability company of the State of Delaware | Methods and systems for speech adaptation data |
US20150116096A1 (en) * | 2013-03-13 | 2015-04-30 | Panasonic Corporation | Information apparatus, apparatus to be controlled, radio communication system, and method for selecting apparatus to be controlled |
US20160005404A1 (en) * | 2014-07-01 | 2016-01-07 | Panasonic Intellectual Property Corporation Of America | Device control method and electric device |
US20160156508A1 (en) * | 2014-12-01 | 2016-06-02 | Gwangju Institute Of Science And Technology | Integrated network controlling system, network controlling apparatus and method, and a readable medium containing the method |
US20160259656A1 (en) * | 2015-03-08 | 2016-09-08 | Apple Inc. | Virtual assistant continuity |
US20170061784A1 (en) * | 2015-08-27 | 2017-03-02 | Altorr Corporation | Sip-and-puff Controller of Electrical Devices for Paraplegic or Quadriplegic Humans |
Also Published As
Publication number | Publication date |
---|---|
JP6482911B2 (en) | 2019-03-13 |
EP2963630A2 (en) | 2016-01-06 |
EP2963630A3 (en) | 2016-03-16 |
US20160005404A1 (en) | 2016-01-07 |
JP2016027688A (en) | 2016-02-18 |
US9721572B2 (en) | 2017-08-01 |
EP2963630B1 (en) | 2017-09-13 |
EP3252730A1 (en) | 2017-12-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9721572B2 (en) | Device control method and electric device | |
AU2020200103B2 (en) | Intelligent assistant for home automation | |
US10068571B2 (en) | Voice control method and voice control system | |
US10991374B2 (en) | Request-response procedure based voice control method, voice control device and computer readable storage medium | |
KR102429436B1 (en) | Server for seleting a target device according to a voice input, and controlling the selected target device, and method for operating the same | |
US20220036882A1 (en) | Electronic apparatus, system and method for using speech recognition service | |
CN109343819B (en) | Display apparatus and method for controlling display apparatus in voice recognition system | |
US20180096683A1 (en) | Processing Voice Commands Based on Device Topology | |
US20120316876A1 (en) | Display Device, Method for Thereof and Voice Recognition System | |
US20170133013A1 (en) | Voice control method and voice control system | |
KR20140089871A (en) | Interactive server, control method thereof and interactive system | |
WO2019213443A1 (en) | Audio analytics for natural language processing | |
KR102392087B1 (en) | Remotely controller and method for receiving a user's voice thereof | |
EP2557565A1 (en) | Voice recognition method and apparatus | |
JPWO2014199596A1 (en) | Speaker identification method, speaker identification device, and speaker identification system | |
US20140123185A1 (en) | Broadcast receiving apparatus, server and control methods thereof | |
KR20140089836A (en) | Interactive server, display apparatus and controlling method thereof | |
JP2016014967A (en) | Information management method | |
EP3220388A1 (en) | Method for correcting false recognition contained in recognition result of speech of user | |
US11250850B2 (en) | Electronic apparatus and control method thereof | |
JPWO2017175442A1 (en) | Information processing apparatus and information processing method | |
KR20170004497A (en) | Method for controlling various electronic devices or allowing the various electronic devices to be controlled by deep-learning with respect to remocon signal and voice command in use of cloud database | |
US20220122600A1 (en) | Information processing device and information processing method | |
CN112700770A (en) | Voice control method, sound box device, computing device and storage medium | |
CN113314115A (en) | Voice processing method of terminal equipment, terminal equipment and readable storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
AS | Assignment |
Owner name: SOVEREIGN PEAK VENTURES, LLC, TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PANASONIC INTELLECTUAL PROPERTY CORPORATION OF AMERICA;REEL/FRAME:048830/0085 Effective date: 20190308 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |