US20180190299A1 - Data structure pooling of voice activated data packets - Google Patents
Data structure pooling of voice activated data packets Download PDFInfo
- Publication number
- US20180190299A1 US20180190299A1 US15/395,707 US201615395707A US2018190299A1 US 20180190299 A1 US20180190299 A1 US 20180190299A1 US 201615395707 A US201615395707 A US 201615395707A US 2018190299 A1 US2018190299 A1 US 2018190299A1
- Authority
- US
- United States
- Prior art keywords
- data structure
- action
- computing device
- pooling
- parameter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000011176 pooling Methods 0.000 title claims abstract description 125
- 230000009471 action Effects 0.000 claims abstract description 325
- 230000005236 sound signal Effects 0.000 claims abstract description 106
- 238000000034 method Methods 0.000 claims abstract description 62
- 238000012545 processing Methods 0.000 claims description 192
- 230000004044 response Effects 0.000 claims description 45
- 230000005540 biological transmission Effects 0.000 claims description 27
- 238000004891 communication Methods 0.000 description 14
- 238000006243 chemical reaction Methods 0.000 description 13
- 238000004590 computer program Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 11
- 238000013515 script Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000000670 limiting effect Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 3
- 230000000644 propagated effect Effects 0.000 description 3
- 238000012384 transportation and delivery Methods 0.000 description 3
- 239000003795 chemical substances by application Substances 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000000153 supplemental effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- IRLPACMLTUPBCL-KQYNXXCUSA-N 5'-adenylyl sulfate Chemical compound C1=NC=2C(N)=NC=NC=2N1[C@@H]1O[C@H](COP(O)(=O)OS(O)(=O)=O)[C@@H](O)[C@H]1O IRLPACMLTUPBCL-KQYNXXCUSA-N 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000001816 cooling Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000013439 planning Methods 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2428—Query predicate definition using graphical user interfaces, including menus and forms
-
- G06F17/30398—
-
- G06F17/30477—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G10L15/18—Speech classification or search using natural language modelling
- G10L15/1822—Parsing for meaning understanding
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/28—Constructional details of speech recognition systems
- G10L15/30—Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G10L2015/088—Word spotting
Definitions
- Excessive network transmissions, packet-based or otherwise, of network traffic data between computing devices can prevent a computing device from properly processing the network traffic data, completing an operation related to the network traffic data, or timely responding to the network traffic data.
- the excessive network transmissions of network traffic data can also complicate data routing or degrade the quality of the response if the responding computing device is at or above its processing capacity, which may result in inefficient bandwidth utilization.
- the control of network transmissions corresponding to content item objects can be complicated by the large number of content item objects that can initiate network transmissions of network traffic data between computing devices.
- At least one aspect is directed to a system to manage voice activated threads in a voice activated data packet based computer network environment.
- a natural language processor component executed by a data processing system can receive, via an interface of the data processing system, data packets comprising a first input audio signal detected by a sensor of a first client computing device.
- the natural language processor component can parse the first input audio signal to identify a first request and a first trigger keyword corresponding to the first request.
- a direct action application programming interface (“API”) of the data processing system can generate, based on the first trigger keyword and in response to the first request, a first action data structure with a parameter defining a first action.
- API application programming interface
- the natural language processor component can receive, via the interface of the data processing system, data packets comprising a second input audio signal detected by a sensor of a second client computing device, and can parse the second input audio signal to identify a second request and a second trigger keyword corresponding to the second request.
- the direct action API can generate, based on the second trigger keyword and in response to the second request, a second action data structure with a parameter defining a second action.
- a pooling component of the data processing system can determine, based on a heuristic technique applied to the parameter of the first action data structure and the parameter of the second action data structure, a pooling parameter that indicates a level of overlap between the first action data structure and the second action data structure.
- the pooling component can combine, based on the pooling parameter, the first action data structure with the second action data structure into a pooled data structure.
- the pooling component can transmit, via a computer network, the pooled data structure to a service provider computing device to cause the service provider computing device to perform an operation defined by the pooled data structure and corresponding to the first action and the second action.
- At least one aspect is directed to a method to manage voice activated threads in a voice activated data packet based computer network environment.
- the method can include receiving, by a natural language processor component executed by a data processing system, via an interface of the data processing system, data packets comprising a first input audio signal detected by a sensor of a first client computing device.
- the method can include parsing, by the natural language processor component, the first input audio signal to identify a first request and a first trigger keyword corresponding to the first request.
- the method can include generating, by a direct action application programming interface (“API”) of the data processing system, based on the first trigger keyword and in response to the first request, a first action data structure with a parameter defining a first action.
- API application programming interface
- the method can include receiving, by the natural language processor component, via the interface of the data processing system, data packets comprising a second input audio signal detected by a sensor of a second client computing device, and parsing, by the natural language processor component, the second input audio signal to identify a second request and a second trigger keyword corresponding to the second request.
- the method can include generating, by the direct action API, based on the second trigger keyword and in response to the second request, a second action data structure with a parameter defining a second action.
- the method can include determining, by a pooling component of the data processing system, based on a heuristic technique applied to the parameter of the first action data structure and the parameter of the second action data structure, a pooling parameter that indicates a level of overlap between the first action data structure and the second action data structure.
- the method can include combining, based on the pooling parameter, the first action data structure with the second action data structure into a pooled data structure.
- the method can include transmitting, via a computer network, the pooled data structure to a service provider computing device to cause the service provider computing device to perform an operation defined by the pooled data structure and corresponding to the first action and the second action.
- At least one aspect is directed to a computer readable storage medium storing instructions that when executed by one or more data processors, cause the one or more data processors to perform operations to manage voice activated threads in a voice activated data packet based computer network environment.
- the operations can include receiving, by a natural language processor component executed by a data processing system, via an interface of the data processing system, data packets comprising a first input audio signal detected by a sensor of a first client computing device.
- the operations can include parsing, by the natural language processor component, the first input audio signal to identify a first request and a first trigger keyword corresponding to the first request.
- the operations can include generating, by a direct action application programming interface (“API”) of the data processing system, based on the first trigger keyword and in response to the first request, a first action data structure with a parameter defining a first action.
- the operations can include receiving, by the natural language processor component, via the interface of the data processing system, data packets comprising a second input audio signal detected by a sensor of a second client computing device, and parsing, by the natural language processor component, the second input audio signal to identify a second request and a second trigger keyword corresponding to the second request.
- the operations can include generating, by the direct action API, based on the second trigger keyword and in response to the second request, a second action data structure with a parameter defining a second action.
- the operations can include determining, by a pooling component of the data processing system, based on a heuristic technique applied to the parameter of the first action data structure and the parameter of the second action data structure, a pooling parameter that indicates a level of overlap between the first action data structure and the second action data structure.
- the operations can include combining, based on the pooling parameter, the first action data structure with the second action data structure into a pooled data structure.
- the operations can include transmitting, via a computer network, the pooled data structure to a service provider computing device to cause the service provider computing device to perform an operation defined by the pooled data structure and corresponding to the first action and the second action.
- FIG. 1 depicts a system to manage voice activated threads in a voice activated data packet based computer network environment
- FIG. 2 depicts a functional diagram of a pooled data structure for a voice activated data packet based computer network environment
- FIG. 3 depicts a functional diagram of action data structure pooling and communication between remote data processing systems and service provider computing devices in a voice activated data packet based computer network environment
- FIG. 4 depicts method to manage voice activated threads in a voice activated data packet based computer network environment
- FIG. 5 is a block diagram illustrating a general architecture for a computer system that may be employed to implement elements of the systems and methods described and illustrated herein.
- Systems and methods of the present disclosure relate generally to a data processing system that manages redundant or overlapping voice activated threads in a voice activated computer network environment.
- the voice activated threads can indicate one or more actions represented by at least one action data structure.
- the data processing system can improve the efficiency and effectiveness of data packet transmission over one or more computer networks by, for example, pooling (e.g., merging, aggregating, or batching) action data structures due to overlapping subject matter or other overlapping or common parameters.
- Data packets or other protocol based signals corresponding to the pooled action data structures can be processed by a direct action application programming interface (API) of the data processing system to provide responsive data (e.g., audio output) or to accomplish a task or action related to the pooled action data structure.
- the data packets or other protocol based signals corresponding to the pooled action data structures can be routed through a computer network from the data processing system to at least one service provider computing device to accomplish a task or action related to the pooled action data
- the data processing system can reduce processor utilization, power consumption, bandwidth utilization, and memory usage relative to individual or non-pooled evaluation of the individual action data structures. For example, providing the pooled action data structure to the service provider computing device via a computer network can use less bandwidth than providing multiple individual action data structures to the service provider computing device.
- the pooled action data structure can eliminate redundancies present in multiple individual action data structures, resulting in decreased processor and power utilization by the data processing system when determining actions responsive to the pooled action data structure, relative to determining actions responsive to each individual or un-pooled action data structure.
- the data processing system can reduce, delay, or eliminate data processing associated with individual action data structure operations. This saves processing power and other computing resources such as memory, reduces electrical power consumption by the data processing system and the reduced data transmissions via the computer network reduces bandwidth requirements and usage of the data processing system.
- the systems and methods described herein can include a data processing system that receives an input audio query, which can also be referred to as an input audio signal. From the input audio signal the data processing system can identify a request and a trigger keyword corresponding to the request. Based on the trigger keyword or the request, the data processing system can determine a first action data structure that includes at least one parameter defining a first action. The data processing system can also receive a second input audio signal, and from at least one request or trigger keyword of the second input audio signal, can identify from the second input audio signal a second action data structure that includes at least one parameter defining the second action. The data processing system can determine from the parameters of the first and second action data structures, at least one pooling parameter. The pooling parameter can indicate a level of overlap (e.g., subject matter commonality) between the first action data structure and the second action data structure.
- the pooling parameter can indicate a level of overlap (e.g., subject matter commonality) between the first action data structure and the second action data structure.
- the data processing system can combine the first action data structure and the second action data structure to generate a pooled data structure.
- the pooled data structure can indicate at least one parameter of actions (e.g., the first action or the second action) common to client computing devices that are the sources of the input audio signals.
- the data processing system can process the pooled data structure to generate output signals (e.g., an audio content item for rendering at one or more client computing devices), or to accomplish tasks related to the first and second actions.
- the data processing system can also transmit the pooled data structure to at least one service provider computing device, to cause the service provider to perform operations defined by the pooled data structure. These operations can correspond to the first or second actions, and can accomplish tasks related to the first or second actions. Pooling the action data structures causes the operations or tasks that correspond to the first or second actions to be performed or accomplished by the data processing system (or by the service provider computing device) with less data processing and less electrical power usage than would be the case without the pooling operation.
- FIG. 1 depicts an example system 100 to manage or otherwise optimize processing of voice activated threads in a voice activated data packet (or other protocol) based computer network environment.
- the system 100 can include at least one data processing system 105 .
- the data processing system 105 can include at least one server having at least one processor.
- the data processing system 105 can include a plurality of servers located in at least one data center or server farm.
- the data processing system 105 can determine, from audio input signals, requests and trigger keywords associated with the request. Based on the requests and trigger keywords the data processing system 105 can generate action data structures, each having respective parameters that indicate or define respective actions.
- the data processing system 105 can determine, from the respective parameters, a level of commonality or overlap between the action data structures. Responsive to a determination by the data processing system of a sufficient level of overlap, the data processing system 105 can pool (or otherwise combine or merge) multiple different action data structures into a pooled action data structure.
- the data processing system 105 can include multiple, logically-grouped servers and facilitate distributed computing techniques.
- the logical group of servers may be referred to as a data center, server farm or a machine farm.
- the servers can be geographically dispersed.
- a data center or machine farm may be administered as a single entity, or the machine farm can include a plurality of machine farms.
- the servers within each machine farm can be heterogeneous—one or more of the servers or machines can operate according to one or more type of operating system platform.
- the data processing system 105 can include servers in a data center that are stored in one or more high-density rack systems, along with associated storage systems, located for example in an enterprise data center.
- the data processing system 105 with consolidated servers in this way can improve system manageability, data security, physical security of the system, and system performance by locating servers and high performance storage systems on localized high performance networks.
- Centralization of all or some of the data processing system 105 components, including servers and storage systems, and coupling them with advanced system management tools allows more efficient use of server resources, which saves power and processing requirements and reduces bandwidth usage.
- the data processing system 105 can include at least one natural language processor (NLP) component 110 , at least one interface 115 , at least one pooling component 120 , at least one content selector component 125 , at least one audio signal generator component 130 , at least one direct action application programming interface (API) 135 , at least one session handler component 140 , and at least one data repository 145 .
- NLP natural language processor
- API application programming interface
- the NLP component 110 , interface 115 , pooling component 120 , content selector component 125 , audio signal generator component 130 , direct action API 135 , and session handler component 140 can each include at least one processing unit, server, virtual server, circuit, engine, agent, appliance, or other logic device such as programmable logic arrays configured to communicate with the data repository 145 and with other computing devices (e.g., the client computing device 150 , the content provider computing device 155 , or the service provider computing device 160 ) via the at least one computer network 165 .
- the network 165 can include computer networks such as the internet, local, wide, metro or other area networks, intranets, satellite networks, other computer networks such as voice or data mobile phone communication networks, and combinations thereof.
- the network 165 can include or constitute a display network, e.g., a subset of information resources available on the internet that are associated with a voice activated assistant system, content placement or search engine results system, or that are eligible to include third party content items as part of a content item placement campaign.
- the network 165 can be used by the data processing system 105 to access information resources such as web pages, web sites, domain names, or uniform resource locators that can be presented, output, rendered, or displayed by the client computing device 150 .
- information resources such as web pages, web sites, domain names, or uniform resource locators that can be presented, output, rendered, or displayed by the client computing device 150 .
- a user of the client computing device 150 can access information or data provided by the content provider computing device 155 or the service provider computing device 160 .
- the network 165 can include, for example a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communication network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, a SDH (Synchronous Digital Hierarchy) network, a wireless network or a wireline network, and combinations thereof.
- the network 165 can include a wireless link, such as an infrared channel or satellite band.
- the topology of the network 165 may include a bus, star, or ring network topology.
- the network 165 can include mobile telephone networks using any protocol or protocols used to communicate among mobile devices, including advanced mobile phone protocol (“AMPS”), time division multiple access (“TDMA”), code-division multiple access (“CDMA”), global system for mobile communication (“GSM”), general packet radio services (“GPRS”) or universal mobile telecommunications system (“UMTS”).
- AMPS advanced mobile phone protocol
- TDMA time division multiple access
- CDMA code-division multiple access
- GSM global system for mobile communication
- GPRS general packet radio services
- UMTS universal mobile telecommunications system
- Different types of data may be transmitted via different protocols, or the same types of data may be transmitted via different protocols.
- the client computing device 150 , the content provider computing device 155 , and the service provider computing device 160 can each include at least one logic device such as a computing device having a processor to communicate with each other or with the data processing system 105 via the network 165 .
- the client computing device 150 , the content provider computing device 155 , and the service provider computing device 160 can each include at least one server, processor or memory, or a plurality of computation resources or servers located in at least one data center.
- the client computing device 150 , the content provider computing device 155 , and the service provider computing device 160 can each include at least one computing device such as a desktop computer, laptop, tablet, personal digital assistant, smartphone, portable computer, server, thin client computer, virtual server, or other computing device.
- the client computing device 150 can include at least one sensor 151 , at least one transducer 152 , at least one audio driver 153 , and at least one speaker 154 .
- the sensor 151 can include a microphone or audio input sensor.
- the transducer 152 can convert the audio input into an electronic signal.
- the audio driver 153 can include a script or program executed by one or more processors of the client computing device 150 to control the sensor 151 , the transducer 152 or the audio driver 153 , among other components of the client computing device 150 to process audio input or provide audio output.
- the speaker 154 can transmit or render the audio output signal.
- the client computing device 150 can be associated with an end user that enters voice queries as audio input into the client computing device 150 (via the sensor 151 ) and receives audio output in the form of a computer generated voice that can be provided from the data processing system 105 (or the content provider computing device 155 or the service provider computing device 160 ) to the client computing device 150 , output from the speaker 154 .
- the computer generated voice can include recordings from a real person or computer generated language.
- the content provider computing device 155 can provide audio based content items for display by the client computing device 150 as an audio output content item.
- the content item can include an offer for a good or service, such as a voice based message that states: “Would you like me to order you a taxi?”
- the content provider computing device 155 can include memory to store a series of audio content items that can be provided in response to a voice based query.
- the content provider computing device 155 can also provide audio based content items (or other content items) to the data processing system 105 where they can be stored in the data repository 145 .
- the data processing system 105 can select the audio content items and provide (or instruct the content provider computing device 155 to provide) the audio content items to the client computing device 150 .
- the audio based content items can be exclusively audio or can be combined with text, image, or video data.
- the data processing system 105 can also perform conversions, e.g., accomplish tasks responsive to the input audio signal. For example, via the direct action API 135 , the data processing system 105 can order a taxi responsive to a request to do so received from the client computing device 150 .
- the data processing system 105 can also perform a conversion by providing data (e.g., a pooled data structure) to the service provider computing device 160 so that the service provider computing device can accomplish a task by, for example, ordering a taxi responsive to a request to do so received by the data processing system 105 from the client computing device 150 .
- the data processing system 105 can also obtain conversion information, e.g., indicating that a task or action was accomplished, from the client computing devices 150 , from the content provider computing devices 155 , or from the service provider computing devices 160 .
- the service provider computing device 160 can include at least one service provider natural language processor (NLP) component 161 and at least one service provider interface 162 .
- the service provider NLP component 161 (or other components such as a direct action API of the service provider computing device 160 ) can engage with the client computing device 150 (via the data processing system 105 or bypassing the data processing system 105 ) to create a back-and-forth real-time voice or audio based conversation (e.g., a session) between the client computing device 150 and the service provider computing device 160 .
- the service provider interface 162 can receive or provide data messages to the direct action API 135 of the data processing system 105 .
- the service provider computing device 160 and the content provider computing device 155 can be associated with the same entity.
- the content provider computing device 155 can create, store, or make available content items for a car sharing service, and the service provider computing device 160 can establish a session with the client computing device 150 to arrange for a delivery of a taxi or car of the car share service to pick up the end user of the client computing device 150 .
- the data processing system 105 via the direct action API 135 , the NLP component 110 or other components can also establish the session with the client computing device, including or bypassing the service provider computing device 160 , to arrange for example for a delivery of a taxi or car of the car share service.
- the data repository 145 can include one or more local or distributed databases, and can include a database management system.
- the data repository 145 can include computer data storage or memory and can store one or more parameters 146 , one or more policies 147 , content data 148 , or templates 149 among other data.
- the parameters 146 , policies 147 , and templates 149 can include information such as rules about a voice based session between the client computing device 150 and the data processing system 105 (or the service provider computing device 160 ).
- the content data 148 can include content items for audio output or associated metadata, as well as input audio messages that can be part of one or more communication sessions with the client computing device 150 .
- the system 100 can optimize or manage processing of voice activated threads (e.g., that indicate one or more actions) in a voice activated data packet (or other protocol) environment.
- the data processing system 105 can include or be part of a voice activated assistant service, voice command device, intelligent personal assistant, knowledge navigator, event planning, or other assistant program.
- the data processing system 105 can provide one or more instances of audio output for display from the client computing device 150 to accomplish tasks related to this input audio signal.
- the tasks can include, for example, communicating with the service provider computing device 160 or other third party computing devices to make dinner reservations or purchase movie tickets. For example, an end user can enter an input audio signal into the client computing device 150 of: “OK, I would like to go to go dinner and then a movie tonight.”
- the data processing system 105 can include an application, script or program installed at the client computing device 150 , such as an app of a voice activated assistant system to communicate input audio signals to the interface 115 of the data processing system 105 and to drive components of the client computing device to render output audio signals.
- the data processing system 105 can receive data packets or other signal that includes or identifies an audio input signal.
- the data processing system 105 can execute or run the NLP component 110 to receive the audio input signal.
- the audio input signal can be detected by the sensor 151 (e.g., a microphone) of the client computing device. Via the transducer 152 , the audio driver 153 , or other components the client computing device 150 can provide the audio input signal to the data processing system 105 (e.g., via the network 165 ) where it can be received (e.g., by the interface 115 ) and provided to the NLP component 110 or stored in the data repository 145 as content data 148 .
- the data processing system 105 e.g., via the network 165
- the NLP component 110 can obtain the input audio signal. From the input audio signal, the NLP component 110 can identify at least one request or at least one trigger keyword corresponding to the request.
- the request can indicate intent or subject matter of the input audio signal.
- the trigger keyword can indicate a type of action likely to be taken. For example, the NLP component 110 can parse the input audio signal to identify at least one request to leave home for the evening to attend dinner and a movie.
- the trigger keyword can include at least one word, phrase, root or partial word, or derivative indicating an action to be taken. For example, the trigger keyword “go” or “to go to” from the input audio signal can indicate a need for transport.
- the input audio signal (or the identified request) does not directly express an intent for transport, however the trigger keyword indicates that transport is an ancillary or sub-action to at least one other action that is indicated by the request.
- the NLP component 110 can convert the audio input signal into recognized text by comparing the input signal against a stored, representative set of audio waveforms (e.g., in the data repository 145 ) and choosing the closest matches.
- the representative waveforms are generated across a large set of users, and can be augmented with speech samples.
- the NLP component 110 can match the text to words that are associated, for example via training across users or through manual specification, with actions that the data processing system 105 can serve.
- the direct action API 135 can execute programs or scripts, for example from the NLP component 110 , the pooling component 120 , or the content selector component 125 to identify content items for one or more of these actions.
- the direct action API 135 can execute a specified action to satisfy the end user's intention, as determined by the data processing system 105 .
- the direct action API 135 can execute code or a dialog script that identifies the parameters required to fulfill a user request.
- code can lookup additional information, e.g., in the data repository 145 , such as the name of a home automation service, or it can provide audio output for rendering at the client computing device 150 to ask the end user questions such as the intended destination of a requested taxi, for example.
- the direct action API 135 can determine necessary parameters and can package the information into an action data structure, which can then be sent to another component such as the content selector component 125 or to the service provider computing device 160 to be fulfilled.
- the direction action API 135 can execute on, or include, hardware of the data processing system 105 such as one or more processors of one or more servers in one or more data centers.
- the direct action API 135 of the data processing system 105 can generate, based on the request or the trigger keyword, data structures for the actions of the thread.
- the action data structures can be generated responsive to the request.
- the action data structure can be included in the messages that are transmitted to or received by the service provider computing device 160 , e.g. as part of or associated with pooled data structures.
- the direct action API 135 can determine to which of a plurality of service provider computing devices 160 the message should be sent. For example, if an input audio signal includes “order a taxi,” the NLP component 110 can identify the trigger word “order” and the request for a taxi.
- the direct action API 135 can package the request into an action data structure for transmission as a message to a service provider computing device 160 of a taxi service.
- the message can also be passed to the content selector component 125 .
- the action data structure can include information for completing the request. In this example, the information can include a pick up location and a destination location.
- the direct action API 135 can retrieve a template 149 from the repository 145 to determine which fields to include in the action data structure.
- the direct action API 135 can retrieve content from the repository 145 to obtain information for the fields of the data structure.
- the direct action API 135 can populate the fields from the template with that information to generate the data structure.
- the direct action API 135 can also populate the fields with data from the input audio signal.
- the templates 149 can be standardized for categories of service providers or can be standardized for specific service providers.
- ride sharing service providers can use the following standardized template 149 to create the data structure: ⁇ client_device_identifier; authentication_credentials; pick_up_location; destination_location; no_passengers; service_level ⁇ .
- the NLP component 110 can obtain the input audio signal as data packet or other protocol based transmissions from the client computing device 150 .
- the input audio signal could be a voice input into the sensor 152 (e.g., a microphone) of “OK, take me home”.
- the NLP component 110 can identify at least one request or at least one trigger keyword.
- the request can indicate subject matter of the input audio signal, e.g., travel.
- the trigger keyword can indicate an action, e.g., “take” or “home” can indicate a need for a car share service to a destination associated with a home destination of the client computing device 150 that originated the input audio signal.
- the direct action API 135 can obtain the requests or trigger keywords that correspond to the input audio signal, and from this information can generate an action data structure.
- the action data structure can include information used, for example, to communicate with the service provider computing device to reserve a car from a car share service to take the end user (who originated the input audio signal into the client computing device) to a destination.
- the action data structure can, with end user consent, include parameters such as information identifying the location of the client computing device 150 to arrange the pickup by the car share service, as well as information identifying the destination location.
- the parameters can be stored in and retrieved from the data repository (e.g., parameters 146 ) by data processing system 105 components.
- the direction action API 135 can also generate the action data structure to include or indicate parameters such as account information of an account associated with the car share service.
- the direct action API 135 can obtain, with end user consent, this account information (or other parameter) from an app or program for the car share service that is installed on the client computing device 150 .
- the parameters can indicate or define actions associated with the action data structures.
- the direct action API 135 can be installed on the data processing system 105 , remote from the client computing device 150 .
- the direct action API 135 can perform a direct conversion, e.g., place an order with the service provider computing device 160 for the car share service trip, or indirectly by providing the action data structure to the service provider computing device 160 for the service provider computing device 160 to process the action data structure in order to complete the car reservation transaction.
- the NLP component 110 can obtain multiple input audio signals from the same or different client computing devices 150 . For each input audio signal, the NLP component can identify or create corresponding requests or corresponding trigger keywords. The direct action API 135 can generate, based on the respective requests or keywords, action data structures corresponding to the respective input audio signals. For example, the NLP component 110 can receive a first input audio signal indicating the voice input “OK, take me home”. In response, the direct action API 135 can generate a first action data structure that can be used to reserve a car from a car share service for the end user associated with the first input audio signal. The NLP component 110 can also receive a second input audio signal indicating, for example, “OK, can I get a taxi please”. In response, the direct action API 135 can generate a second action data structure for a car share service for the end user associated with the second input audio signal.
- the first and second input audio signal can originate from the same client computing device 150 or from different client computing devices 150 , and can be associated with the same or different end users.
- the NLP component 110 can receive first input audio signal from a first client computing device 150 and can receive a second input audio signal from a second client computing device 150 that can be associated with different end users. From this information, which can include metadata or header information of packet based transmissions, among supplemental information that can be obtained from sources other than the input audio signal, such as from content data 148 of the data repository 145 , the direct action API 135 can generate a first data structure for the first input audio signal and a second data structure for the second input audio signal.
- the first and second data structures can both indicate requests to reserve a car from a car share service.
- Each action data structure can include parameters that indicate their own origination and destination locations, or can indicate different accounts that different end users have with the car share service (or other parameters).
- the action data structure can also include one or more parameters that indicate that a client computing device 150 (or associated end user) does not have an account with the car share service or other entity.
- the pooling component 120 can identify overlap or similarities between two or more different action data structures that were generated from different input audio signals.
- one or more processors of the data processing system 105 can include or execute the pooling component 120 , remote from the client computing device 105 and remote from the service provider computing device 160 , to identify overlapping or common subject matter of two or more different action data structures.
- the pooling component 120 can apply a heuristic technique to identify with a threshold degree of certainty of common patterns among the parameters of the different action data structures.
- the level of overlap can be or include a similarity metric that indicates a common feature of multiple action data structures.
- the pooling component 120 can determine a similarity metric indicating the multiple action data structures relate to time data, location data, service provider entity data, subject matter data, or geographic data. For example, the pooling component 120 can determine that multiple action data structures include requests for transport from the same transportation company, or reservations for different times or evenings for the same restaurant, requests to travel to or from a common destination or geographic area, or requests to order one or more products from one entity.
- the pooling component 120 can identify the similarity metrics or common features as at least one pooling parameter, and based on the pooling parameter the pooling component can create the batched or pooled data structure.
- the overlap may but need not include at least partially identical code or partially identical action data structures.
- the pooling component 120 can identify overlap based on requests for services from the same entity, or based on commonality among parameters of the action data structures such as a similar, but not identical, time or location data.
- parameters of the first action data structure can indicate a request for a car share service from a pickup location of ‘point A’ to a destination of ‘point B’.
- Parameters of the second action data structure can indicate a different request for a car share service from a pickup location of ‘point C’ to a destination of ‘point D’.
- the data processing system 105 or component thereof e.g., the pooling component 120
- the data processing system 105 or component thereof can determine that pickup location ‘point C’ is located between, or within a threshold distance (e.g., 1 mile) of a route between, ‘point A’ and ‘point B’.
- the pooling component 120 can also determine that ‘point D’ is located within a threshold distance (e.g., 5 miles) of destination ‘point B’. While the parameters of the different action data structures (e.g., parameters indicating different points A-D, or other information such as different accounts associated with different client computing devices 150 , the pooling component 120 can determine that sufficient overlap or commonality exists between the parameters due, for example to route similarities satisfying threshold metrics. The pooling component 120 can also determine that the two action data structures overlap because their respective parameters indicate that they are associated with respective input audio signals that were made within a threshold time period of one another, such as within 1 minute, one hour, of other time period of each other, or because their respective parameters indicate that they are both requesting a good or service from the same entity.
- a threshold distance e.g., 5 miles
- the pooling component 120 can generate, based on the analysis of at least two action data structures, at least one pooling parameter.
- the pooling parameter can indicate common subject matter, features, entities, or parameters from among two or more action data structures.
- the pooling parameter can indicate that the requests for car share services indicated by parameters of the first and second action data structures share similar locations, or similar routes between respective sources and destinations.
- the direction action API 135 (or the service provider computing device 160 ) can use this information to service the actions associated with or indicated by the action data structures, such as car service rides between respective sources and destinations.
- the pooling component 120 can create at least one pooled data structure 200 .
- the pooled data structure can include a packet based architecture (or other format) and can include a header 205 .
- the header 205 can include supplemental data identifying the data processing system 105 , source or destination address information, governing protocol information, IP addresses, frame synchronization data, size data, resolution data, or metadata associated with action data structures.
- the payload or body of the pooled data structure 200 can include at least one pooling parameter 210 , or other information such as at least one first parameter 215 or at least one second parameters 220 .
- the first parameter 215 can include a parameter of the first action data structure 225
- the second parameters 220 can include a parameter of the second action data structure 230 .
- the first parameter 215 , second parameter 220 , first action data structure 225 and the second action data structure 230 may but need not be included in the pooled data structure 200 .
- the body of the pooled data structure 200 can include only the pooling parameter 210 (and not include any of the first parameter 215 , the second parameter 220 , the first action data structure 225 or the second action data structure 230 ), or the body of the pooled data structure 200 can include the pooling parameter 210 as well as one or more of the first parameter 215 , the second parameter 220 , the first action data structure 225 or the second action data structure 230 .
- the pooled data structure 200 can include one or more individual packets transmitted separately in sequence or parallel as part of one or more data transmissions between the data processing system 105 and the service provider computing device 160 .
- the pooling parameter, or information in headers 205 can indicate that the separate transmissions or separate data packets are associated with the same batch transmission, e.g., the same overall pooled data structure 200 .
- the data processing system 105 can receive, from a first client computing device 150 , the first input audio signal 305 , (e.g., “OK, take me home”).
- the data processing system 105 can also receive, from a second client computing device 150 , the second input audio signal 310 , (e.g., “OK, can I get a taxi please”).
- the data processing system 105 can determine respective parameters that define actions associated with the first input audio signal 305 and the second audio input signal 305 .
- Data processing system 105 components such as the pooling component 120 can apply statistical or heuristic techniques to the parameters associated with the first input audio signal 305 and the second input audio signal 310 to generate at least one pooling parameter indicative of subject matter overlap or commonality between the first input audio signal 305 and the second input audio signal 310 .
- the pooling parameter can indicate that the first input audio signal 305 and the second input audio signal both relate to requests to hire a car from a car share service.
- the data processing system 105 can batch or pool the action data structures corresponding to the first input audio signal 305 and the second input audio signal 310 to create the batched or pooled data structure 200 .
- the data processing system 105 can transmit or otherwise provide the pooled data structure 200 to the service provider computing device 160 for processing by the service provider computing device 160 to reserve at least one car responsive to the first and second input audio signals 305 , 310 .
- the pooled data structure 200 can include pooling parameters generated from more than two action data structures (or more than two input audio signals). For example, a third audio input signal can cause the data processing system 105 to generate a third action data structure for a third action.
- the pooling component 120 can identify at least one pooling parameter based on parameters of the third action data structure. Responsive to this determination, the pooling component 120 can modify the pooled data structure 200 to include or indicate an association with the third data structure, or can include a new pooled data structure associated with three (or more) input audio signals.
- the data processing system 105 can reduce the number, size, or frequency of data transmissions via the network 165 . This can reduce bandwidth utilization of network transmissions, and can reduce processing power and memory storage requirements of both the data processing system 105 (e.g., data repository 145 ) and of the service provider computing device 160 that would otherwise be used to process and transmit individual action data structures that may have a level of redundancy or subject matter overlap contained within their data.
- the pooled data structure 200 may but need not be a single data structure.
- the pooled data structure can include multiple individual data structures transmitted by the data processing system 105 , with or without the pooling parameters, as part of a batched or periodic transmission to one or more service provider computing devices 160 controlled by one entity, such as a car share service provider.
- the data processing system 105 and the service provider computing device 160 can correspond to different entities.
- the data processing system 105 (and components such as the direct action API 135 ) can be part of a voice based assistant computer system designed to interface with end users of the client computing device 150 to provide audio responses to audio input signals.
- the audio signal generator component 130 can create an audio response of “There is a taxi 5 minutes away, would you like to order it?” to the audio input signal of “OK, take me home”.
- the session handler component 140 can establish a communication session via the computer network 165 between the data processing system 105 and the client computing device 150 to provide this audio response.
- the audio response can cause the audio driver 153 of the client computing device 150 to cause the speaker 154 to output acoustic waves to render the audio response to the end user.
- the service provider computing device 160 can be a computing device of the car share or taxi service entity.
- the data processing system 105 processes the input audio signals 305 , 310 to generate the pooled data structure 200 .
- the audio input signals 305 , 310 correspond to actions for the service provider computing device 160 , e.g., ordering a car from the car share service entity that controls the service provider computing device 160 .
- the data processing system 105 can operate as an intermediary appliance or device that can hook, intercept, or otherwise receive the input audio signals that include requests for action by the service provider computing device 160 .
- the data processing system 105 can batch or consolidate these requests into at least one pooled data structure 200 that is provided to the service provider computing device 160 .
- the service provider computing device 160 instead receives a pooled data structure 200 (e.g., a batched transmission of consolidated action data structures) from the intermediary data processing system 105 .
- requests from the client computing devices 150 for goods or services of the service provider computing device 160 are routed through the data processing system 105 , where they are consolidated into a pooled data structure 200 that can be transmitted to the service provider computing device 160 .
- the service provider computing device 160 can perform at least one operation defined or indicated by the pooled data structure 200 .
- the operation can correspond to one or more of the respective actions of the action data structures of the first or second input audio signals 305 , 310 .
- the operations can include reserving a car from a car share service.
- the service provider computing device 160 can provide responses to the client computing devices 150 .
- the service provider computing device 160 can provide a first response 315 to the first client computing device 150 , and can provide a second response 320 to the second client computing device 150 .
- the first and second responses 315 , 320 can bypass the data processing system 105 during transmission from the service provider computing device 160 to the first and second client computing devices 150 .
- the responses 315 can include text, image, or other data messages (including audio) that confirm a conversion action, such as a reservation for a car from a car service.
- the data processing system 105 can receive an indication of the conversion (or of the responses 315 , 320 ) via the computer network 165 from the service provider computing device 160 or from the client computing devices 150 .
- the data processing system 105 can also receive the responses 315 , 320 , from the service provider computing device(s) 160 .
- the service provider entity that corresponds to one or more service provider computing devices 160 can perform multiple operations to accomplish different actions indicated by the pooled data structure 200 .
- the service provider entity can provide grocery deliveries and operate a car share service for human transport.
- the pooled data structure 200 can indicate both of these actions, which can be accomplished by a single service provider entity that controls one or more service provider computing devices 160 .
- the service provider computing devices 160 can also provide pooling policy data to the data processing system 105 .
- the direct action API 135 can receive from the service provider computing devices 160 pooling policy data.
- the pooling policy data can provide policy guidelines used by the pooling component 120 to generate the pooled data structure 200 .
- the pooling policy data can be stored in the data repository 145 (e.g., as policies 147 ).
- the pooling policy data can indicate a format of the pooled data structure, template information (e.g., template 149 ) or information size data, or about when, how, how frequently, or what information should be included in the pooled data structure.
- the pooling component 120 can create the pooled data structure 200 in compliance with the pooling policy data.
- the responses 315 , 320 include the same substantive response transmitted separately to both the first and second client computing devices 150 .
- the service provider computing device 160 can consolidate the first and second actions, indicated by respective action data structures in the pooled data structure 200 and can provide a single car the arrives in sequence at pickup locations associated with the first and second client computing devices 150 .
- the responses 315 , 320 need not be pooled.
- the responses 315 , 320 can be different from each other, or customized responses to the first action data structure 225 or the second action data structure 230 (or the first and second parameters 215 , 220 ) of the pooled data structure 200 .
- the service provider computing devices 160 can provide the response 315 with information about a first car share reservation for the first client computing device 150 , and can provide the response 320 with information about a different, second car share reservation for the second client computing device 150 .
- the pooling parameter created by the pooling component 120 and used to generate the pooled data structure 200 can also include temporal, destination, or location data.
- the pooling parameter can indicate a plurality of action data structures that each correspond to requests for services from a single entity associated with the service provider computing device (e.g., destination information), such as reservation requests for a particular restaurant, orders for a good manufactured by the entity, or car share reservation requests from a car share service.
- the pooling parameter can also indicate a number of action data structures made within a defined time period, such as within a 2 minute interval each requesting a car service from the same car share entity.
- the pooling parameter can include location information, such as location information identifying a defined geographical area (e.g., a city block or neighborhood) from which a plurality of people provide input audio signals requesting a service from the same entity associated with the service provider computing device 160 .
- location information such as location information identifying a defined geographical area (e.g., a city block or neighborhood) from which a plurality of people provide input audio signals requesting a service from the same entity associated with the service provider computing device 160 .
- the pooling component 120 can generate the pooled data structure 200 and provide it to the service provider computing device 160 , which can perform operations responsive to the pooled data structure, and provide associated responses 315 , 320 , to the client computing devices 150 .
- the pooled data structure 200 can include only operations for a single entity, which is a different entity than the entity associated with the data processing system 105 .
- the data processing system 105 (or component such as the pooling component 120 ) can provide a first pooled data structure to one or more first service provider computing devices 160 controlled by a first entity, and can generate and provide a second pooled data structure to one or more second service provider computing devices 160 that are controlled by a different, second entity unrelated to the first entity.
- the first pooled data structure can relate to consolidated requests to buy tickets for a local sporting event from a ticket broker company
- the second pooled data structure can relate to requests to make travel reservations for an overseas tropical beach vacation from a travel agent company.
- the content selector component 125 can obtain indications of at least one action of the action data structures of the pooled action data structure 200 . From the information received by the content selector component 125 , e.g., an indication of an action, the content selector component 125 can identify at least one content item.
- the content item can be responsive or related to the third action.
- the content item can include an audio message offering services of a car share company, responsive to the third action that indicates a transportation need.
- the content selector component 125 can query the data repository 145 to select or otherwise identify the content item, e.g., from the content data 148 .
- the content selector component 125 can also select the content item from the content provider computing device 155 .
- the content provider computing device 155 can provide a content item to the data processing system 105 (or component thereof) for eventual output by the client computing device 150 .
- the content item can relate to a service provider associated with the action data structure, e.g., an identified car share service, or another service such as a competitor or ancillary service.
- the audio signal generator component 130 can generate or otherwise obtain an output signal that includes the content item.
- the data processing system 105 can execute the audio signal generator component to generate or create an output signal corresponding to the content item.
- the interface 115 of the data processing system 105 can provide transmit one or more data packets that include the output signal via the computer network 165 to the client computing device 150 .
- the interface 115 can be designed, configured, constructed, or operational to receive and transmit information using, for example, data packets.
- the interface 115 can receive and transmit information using one or more protocols, such as a network protocol.
- the interface 115 can include a hardware interface, software interface, wired interface, or wireless interface.
- the interface 115 can facilitate translating or formatting data from one format to another format.
- the interface 115 can include an application programming interface that includes definitions for communicating between various components, such as software components of the system 100 .
- the data processing system 105 can provide the output signal (e.g., corresponding to the content item or responsive to the input audio signal), from the data repository 145 or from the audio signal generator component 130 to the client computing device 150 .
- the data processing system 105 can also instruct, via data packet transmissions, the content provider computing device 155 or the service provider computing device 160 to provide the output signal to the client computing device 150 .
- the output signal can be obtained, generated, transformed to or transmitted as one or more data packets (or other communications protocol) from the data processing system 105 (or other computing device) to the client computing device 150 .
- the content selector component 125 can select the content item as part of a real-time content selection process.
- the content item can be provided to the client computing device 150 as audio output in a conversational manner in direct response to the input audio signal.
- the real-time content selection process to identify the content item and provide the content item to the client computing device 150 can occur within one minute or less from the time of the input audio signal and be considered real-time.
- the output signal that corresponds to the content item can cause the client computing device 150 to execute the audio driver 153 to drive the speaker 154 to generate an acoustic wave corresponding to the output signal.
- the acoustic wave can include words of or corresponding to the content item for the third action.
- the acoustic wave can be output from the client computing device 150 , e.g., as an acoustic wave that include the audio output of “Your taxi is confirmed” or “While you are waiting for your taxi, there is a coffee shop around the corner”.
- the data processing system 105 can prevent or delay transmission of the pooled data structure to the service provider computing device 160 .
- the interface 115 or a script executing via the direct action API 135 or the pooling component 120 can prevent transmission of data packets that include the pooled data structure until a triggering event.
- the triggering event can include expiration of a pre-determined time period, from less than one minute to a period of days.
- the triggering event can also be based on available bandwidth, processing, or memory.
- the service provider computing device 160 can prompt or query the data processing system for the pooled data structure, which the data processing system 105 transmits in response to the prompt.
- the data processing system 105 can also delay transmission of the pooled data structure to optimize processing utilization. For example, the data processing system 105 can delay content item transmission until an off-peak or non-peak period of data center usage, which results in more efficient utilization of the data center by reducing peak bandwidth usage, heat output or cooling requirements.
- the data processing system 105 can also initiate a conversion or other activity associated with the content item, such as ordering a car service, based on data center utilization rates or bandwidth metrics or requirements of the network 165 or of a data center that includes the data processing system 105 . This can reduce network traffic, load requirements, and save processing and electrical power.
- the data processing system 105 can receive a response to a content item such as an offer for a ride share service that states “would you like a ride home from the movie theater?”
- the response can include an input audio signal, e.g., “yes please” that was entered into the client computing device 150 by the end user and received by the data processing system 105 .
- the direct action API 135 can communicate with the service provider computing device 160 (that can be associated with the content item, such as a car share company) to order a taxi or ride share vehicle for the location of the movie theater at the time the movie ends.
- the data processing system 105 can obtain this location or time information as part of the data packet (or other protocol) based data message communication with the client computing device 150 , from the data repository 145 , or from other sources such as the service provider computing device 160 or the content provider computing device 155 . Confirmation of this order (or other conversion) can be provided as an audio communication from the data processing system 105 to the client computing device 150 in the form of an output signal from the data processing system 105 that drives the client computing device 150 to render audio output such as, “great, you will have a car waiting for you at 11 pm outside the theater.”
- the data processing system 105 via the direct action API 135 , can communicate with the service provider computing device 160 to confirm the order for the car.
- the data processing system 105 can obtain the response (e.g., “yes please”) to the content item (“would you like a ride home from the movie theater?”) and can route a packet based data message to the service provider NPL component 161 (or other component of the service provider computing device). This packet based data message can cause the service provider computing device 160 to effect a conversion, e.g., to make a car pick up reservation outside the movie theater.
- the data processing system 105 can obtain a notification of the conversion, e.g., from the service provider computing device 160 , the content provider computing device 155 , or the client computing device 150 .
- the data processing system 105 can initiate a conversion or action. For example, processors of the data processing system 105 can invoke the direct action API 135 to execute scripts that order a car from a car share service.
- the direct action API 135 can obtain content data 148 (or parameters 146 or policies 147 ) from the data repository 145 , as well as data received with end user consent from the client computing device 150 to determine location, time, user accounts, logistical or other information in order to reserve a car from the car share service.
- the data processing system 105 can also communicate with the service provider computing device 160 to complete the conversion by in this example making the car share pick up reservation.
- the direct action API 135 can initiate the conversion or activity to accomplish an action associated with the pooled data structure (e.g., an action defined by one or more parameters of an action data structure indicated by the pooled data structure)
- FIG. 4 depicts method 400 to manage voice activated threads in a voice activated data packet based computer network environment.
- the method 400 can receive at least one first audio input signal (ACT 405 ).
- the NLP component can receive data packets (ACT 405 ).
- the data processing system 105 can execute, launch, or invoke the NLP component 110 to receive packet or other protocol based transmissions via the network from a first client computing device 150 .
- the data packets can include or correspond to an input audio signal detected by the sensor 151 of a first client computing device 150 , such as a first end user saying “OK, take me home” into the client computing device 150 , e.g., smartphone.
- the method 400 can parse the input audio signal to identify at least one first request or at least one first trigger keyword from the input audio signal (ACT 410 ).
- the NLP component 110 can parse the input audio signal to identify requests (to go “home”) as well as trigger keywords (“take”) that correspond or relate to the request.
- the method 400 can generate at least one first action data structure (ACT 415 ) based on the identified requests or keywords associated with the received first audio input signal.
- the direct action API 135 can generate an action data structure that includes at least one parameter defining a first action, such as a location for a taxi pick-up, a requested service provider, or time information.
- the method 400 can receive at least one second audio input signal (ACT 420 ).
- the NLP component 110 can receive packet or other protocol based transmissions via the network from a second client computing device 150 .
- the data packets can include or correspond to an input audio signal detected by the sensor 151 of the second client computing device 150 , such as an end user saying “OK, can I get a taxi please” into the second client computing device 150 .
- the method 400 can identify at least one second request or at least one second trigger keyword from the second input audio signal (ACT 425 ).
- the NLP component 110 can parse the input audio signal to identify requests (to go “taxi”) as well as trigger keywords (“get”) that correspond or relate to the request.
- the method 400 can generate at least one second action data structure (ACT 430 ) based on the identified requests or keywords associated with the received first audio input signal.
- the direct action API 135 can generate an action data structure that includes at least one parameter defining a second action, such as a location for a taxi pick-up, a requested service provider, or time information.
- the method 400 can determine at least one pooling parameter (ACT 435 ). For example, based on parameters of respective actions that indicate a common requested service provider or other commonality, (e.g., a level of overlap between action data structures), the pooling component 120 can determine a pooling parameter that indicates the common requested service provider.
- the method 400 can generate a pooled data structure (ACT 440 ). For example, based on the indication of the common requested service provider the pooling component 120 can generate at least one pooled data structure that indicates the first and second action data structures, or their associated actions or parameters (ACT 440 ).
- the method 400 can transmit the pooled data structure (ACT 445 ). For example, the data processing system 105 can provide the pooled data structure to one or more service provider computing devices 160 controlled by the common requested service provider entity.
- FIG. 5 is a block diagram of an example computer system 500 .
- the computer system or computing device 500 can include or be used to implement the system 100 , or its components such as the data processing system 105 .
- the computing system 500 includes a bus 505 or other communication component for communicating information and a processor 510 or processing circuit coupled to the bus 505 for processing information.
- the computing system 500 can also include one or more processors 510 or processing circuits coupled to the bus for processing information.
- the computing system 500 also includes main memory 515 , such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 505 for storing information, and instructions to be executed by the processor 510 .
- the main memory 515 can be or include the data repository 145 .
- the main memory 515 can also be used for storing position information, temporary variables, or other intermediate information during execution of instructions by the processor 510 .
- the computing system 500 may further include a read only memory (ROM) 520 or other static storage device coupled to the bus 505 for storing static information and instructions for the processor 510 .
- ROM read only memory
- a storage device 525 such as a solid state device, magnetic disk or optical disk, can be coupled to the bus 505 to persistently store information and instructions.
- the storage device 525 can include or be part of the data repository 145 .
- the computing system 500 may be coupled via the bus 505 to a display 535 , such as a liquid crystal display, or active matrix display, for displaying information to a user.
- a display 535 such as a liquid crystal display, or active matrix display, for displaying information to a user.
- An input device 530 such as a keyboard including alphanumeric and other keys, may be coupled to the bus 505 for communicating information and command selections to the processor 510 .
- the input device 530 can include a touch screen display 535 .
- the input device 530 can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 510 and for controlling cursor movement on the display 535 .
- the display 535 can be part of the data processing system 105 , the client computing device 150 or other component of FIG. 1 , for example.
- the processes, systems and methods described herein can be implemented by the computing system 500 in response to the processor 510 executing an arrangement of instructions contained in main memory 515 .
- Such instructions can be read into main memory 515 from another computer-readable medium, such as the storage device 525 .
- Execution of the arrangement of instructions contained in main memory 515 causes the computing system 500 to perform the illustrative processes described herein.
- One or more processors in a multi-processing arrangement may also be employed to execute the instructions contained in main memory 515 .
- Hard-wired circuitry can be used in place of or in combination with software instructions together with the systems and methods described herein. Systems and methods described herein are not limited to any specific combination of hardware circuitry and software.
- the users may be provided with an opportunity to control whether programs or features that may collect personal information (e.g., information about a user's social network, social actions or activities, a user's preferences, or a user's location), or to control whether or how to receive content from a content server or other data processing system that may be more relevant to the user.
- personal information e.g., information about a user's social network, social actions or activities, a user's preferences, or a user's location
- certain data may be anonymized in one or more ways before it is stored or used, so that personally identifiable information is removed when generating parameters.
- a user's identity may be anonymized so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, postal code, or state level), so that a particular location of a user cannot be determined.
- location information such as to a city, postal code, or state level
- the user may have control over how information is collected about him or her and used by the content server.
- the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
- the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more circuits of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, data processing apparatuses.
- the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
- a computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. While a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices).
- the operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.
- data processing system “computing device” “component” or “data processing apparatus” encompass various apparatuses, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing.
- the apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
- the apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them.
- the apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.
- the direct action API 135 , content selector component 125 , pooling component 120 or NLP component 110 and other data processing system 105 components can include or share one or more data processing apparatuses, systems, computing devices, or processors.
- a computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment.
- a computer program can correspond to a file in a file system.
- a computer program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).
- a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
- the processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs (e.g., components of the data processing system 105 ) to perform actions by operating on input data and generating output.
- the processes and logic flows can also be performed by, and apparatuses can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
- Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.
- semiconductor memory devices e.g., EPROM, EEPROM, and flash memory devices
- magnetic disks e.g., internal hard disks or removable disks
- magneto optical disks e.g., CD ROM and DVD-ROM disks.
- the processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
- the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or a combination of one or more such back-end, middleware, or front-end components.
- the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network.
- Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).
- LAN local area network
- WAN wide area network
- inter-network e.g., the Internet
- peer-to-peer networks e.g., ad hoc peer-to-peer networks.
- the computing system such as system 100 or system 500 can include clients and servers.
- a client and server are generally remote from each other and typically interact through a communication network (e.g., the network 165 ).
- the relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
- a server transmits data (e.g., data packets representing a content item) to a client computing device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client computing device).
- Data generated at the client computing device can be received from the client computing device at the server (e.g., received by the data processing system 105 from the computing device 150 or the content provider computing device 155 or the service provider computing device 160 ).
- the separation of various system components does not require separation in all implementations, and the described program components can be included in a single hardware or software product.
- the NLP component 110 , the content selector component 125 , or the pooling component 120 can be a single component, app, or program, or a logic device having one or more processing circuits, or part of one or more servers of the data processing system 105 .
- references to implementations or elements or acts of the systems and methods herein referred to in the singular may also embrace implementations including a plurality of these elements, and any references in plural to any implementation or element or act herein may also embrace implementations including only a single element.
- References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their components, acts, or elements to single or plural configurations.
- References to any act or element being based on any information, act or element may include implementations where the act or element is based at least in part on any information, act, or element.
- any implementation disclosed herein may be combined with any other implementation or embodiment, and references to “an implementation,” “some implementations,” “one implementation” or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be included in at least one implementation or embodiment. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.
- references to “or” may be construed as inclusive so that any terms described using “or” may indicate any of a single, more than one, and all of the described terms. For example, a reference to “at least one of ‘A’ and ‘B’” can include only ‘A’, only ‘B’, as well as both ‘A’ and ‘B’. Such references used in conjunction with “comprising” or other open terminology can include additional items.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Multimedia (AREA)
- Acoustics & Sound (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Mathematical Physics (AREA)
- Signal Processing (AREA)
- Telephonic Communication Services (AREA)
- Information Transfer Between Computers (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Computer And Data Communications (AREA)
Abstract
Systems and methods of voice activated thread management in a voice activated data packet based environment are provided. A natural language processor (“NLP”) component can receive and parse data packets comprising a first input audio signal to identify a first request and a first trigger keyword. A direct action application programming interface (“API”) can generate a first action data structure with a parameter defining a first action. The NLP component can receive and parse a second input audio signal to identify a second request and a second trigger keyword, and can generate a second action data structure with a parameter defining a second action. A pooling component can generate the first and second action data structures into a pooled data structure, and can transmit the pooled data structure to a service provider computing device to cause it device to perform an operation defined by the pooled data structure.
Description
- Excessive network transmissions, packet-based or otherwise, of network traffic data between computing devices can prevent a computing device from properly processing the network traffic data, completing an operation related to the network traffic data, or timely responding to the network traffic data. The excessive network transmissions of network traffic data can also complicate data routing or degrade the quality of the response if the responding computing device is at or above its processing capacity, which may result in inefficient bandwidth utilization. The control of network transmissions corresponding to content item objects can be complicated by the large number of content item objects that can initiate network transmissions of network traffic data between computing devices.
- At least one aspect is directed to a system to manage voice activated threads in a voice activated data packet based computer network environment. A natural language processor component executed by a data processing system can receive, via an interface of the data processing system, data packets comprising a first input audio signal detected by a sensor of a first client computing device. The natural language processor component can parse the first input audio signal to identify a first request and a first trigger keyword corresponding to the first request. A direct action application programming interface (“API”) of the data processing system can generate, based on the first trigger keyword and in response to the first request, a first action data structure with a parameter defining a first action. The natural language processor component can receive, via the interface of the data processing system, data packets comprising a second input audio signal detected by a sensor of a second client computing device, and can parse the second input audio signal to identify a second request and a second trigger keyword corresponding to the second request. The direct action API can generate, based on the second trigger keyword and in response to the second request, a second action data structure with a parameter defining a second action. A pooling component of the data processing system can determine, based on a heuristic technique applied to the parameter of the first action data structure and the parameter of the second action data structure, a pooling parameter that indicates a level of overlap between the first action data structure and the second action data structure. The pooling component can combine, based on the pooling parameter, the first action data structure with the second action data structure into a pooled data structure. The pooling component can transmit, via a computer network, the pooled data structure to a service provider computing device to cause the service provider computing device to perform an operation defined by the pooled data structure and corresponding to the first action and the second action.
- At least one aspect is directed to a method to manage voice activated threads in a voice activated data packet based computer network environment. The method can include receiving, by a natural language processor component executed by a data processing system, via an interface of the data processing system, data packets comprising a first input audio signal detected by a sensor of a first client computing device. The method can include parsing, by the natural language processor component, the first input audio signal to identify a first request and a first trigger keyword corresponding to the first request. The method can include generating, by a direct action application programming interface (“API”) of the data processing system, based on the first trigger keyword and in response to the first request, a first action data structure with a parameter defining a first action. The method can include receiving, by the natural language processor component, via the interface of the data processing system, data packets comprising a second input audio signal detected by a sensor of a second client computing device, and parsing, by the natural language processor component, the second input audio signal to identify a second request and a second trigger keyword corresponding to the second request. The method can include generating, by the direct action API, based on the second trigger keyword and in response to the second request, a second action data structure with a parameter defining a second action. The method can include determining, by a pooling component of the data processing system, based on a heuristic technique applied to the parameter of the first action data structure and the parameter of the second action data structure, a pooling parameter that indicates a level of overlap between the first action data structure and the second action data structure. The method can include combining, based on the pooling parameter, the first action data structure with the second action data structure into a pooled data structure. The method can include transmitting, via a computer network, the pooled data structure to a service provider computing device to cause the service provider computing device to perform an operation defined by the pooled data structure and corresponding to the first action and the second action.
- At least one aspect is directed to a computer readable storage medium storing instructions that when executed by one or more data processors, cause the one or more data processors to perform operations to manage voice activated threads in a voice activated data packet based computer network environment. The operations can include receiving, by a natural language processor component executed by a data processing system, via an interface of the data processing system, data packets comprising a first input audio signal detected by a sensor of a first client computing device. The operations can include parsing, by the natural language processor component, the first input audio signal to identify a first request and a first trigger keyword corresponding to the first request. The operations can include generating, by a direct action application programming interface (“API”) of the data processing system, based on the first trigger keyword and in response to the first request, a first action data structure with a parameter defining a first action. The operations can include receiving, by the natural language processor component, via the interface of the data processing system, data packets comprising a second input audio signal detected by a sensor of a second client computing device, and parsing, by the natural language processor component, the second input audio signal to identify a second request and a second trigger keyword corresponding to the second request. The operations can include generating, by the direct action API, based on the second trigger keyword and in response to the second request, a second action data structure with a parameter defining a second action. The operations can include determining, by a pooling component of the data processing system, based on a heuristic technique applied to the parameter of the first action data structure and the parameter of the second action data structure, a pooling parameter that indicates a level of overlap between the first action data structure and the second action data structure. The operations can include combining, based on the pooling parameter, the first action data structure with the second action data structure into a pooled data structure. The operations can include transmitting, via a computer network, the pooled data structure to a service provider computing device to cause the service provider computing device to perform an operation defined by the pooled data structure and corresponding to the first action and the second action.
- These and other aspects and implementations are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementations, and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations, and are incorporated in and constitute a part of this specification.
- The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:
-
FIG. 1 depicts a system to manage voice activated threads in a voice activated data packet based computer network environment; -
FIG. 2 depicts a functional diagram of a pooled data structure for a voice activated data packet based computer network environment; -
FIG. 3 depicts a functional diagram of action data structure pooling and communication between remote data processing systems and service provider computing devices in a voice activated data packet based computer network environment; -
FIG. 4 depicts method to manage voice activated threads in a voice activated data packet based computer network environment; and -
FIG. 5 is a block diagram illustrating a general architecture for a computer system that may be employed to implement elements of the systems and methods described and illustrated herein. - Following below are more detailed descriptions of various concepts related to, and implementations of, methods, apparatuses, and systems to manage voice activated threads in a voice activated data packet (or other protocol) based computer network environment. The various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways.
- Systems and methods of the present disclosure relate generally to a data processing system that manages redundant or overlapping voice activated threads in a voice activated computer network environment. The voice activated threads can indicate one or more actions represented by at least one action data structure. The data processing system can improve the efficiency and effectiveness of data packet transmission over one or more computer networks by, for example, pooling (e.g., merging, aggregating, or batching) action data structures due to overlapping subject matter or other overlapping or common parameters. Data packets or other protocol based signals corresponding to the pooled action data structures can be processed by a direct action application programming interface (API) of the data processing system to provide responsive data (e.g., audio output) or to accomplish a task or action related to the pooled action data structure. The data packets or other protocol based signals corresponding to the pooled action data structures can be routed through a computer network from the data processing system to at least one service provider computing device to accomplish a task or action related to the pooled action data structure.
- By merging or pooling multiple action data structures into a single or pooled action data structure, and batch processing the pooled action data structure, the data processing system can reduce processor utilization, power consumption, bandwidth utilization, and memory usage relative to individual or non-pooled evaluation of the individual action data structures. For example, providing the pooled action data structure to the service provider computing device via a computer network can use less bandwidth than providing multiple individual action data structures to the service provider computing device. The pooled action data structure can eliminate redundancies present in multiple individual action data structures, resulting in decreased processor and power utilization by the data processing system when determining actions responsive to the pooled action data structure, relative to determining actions responsive to each individual or un-pooled action data structure.
- By at least partially bypassing individual action data structure processing and instead generating pooled action data structures, the data processing system can reduce, delay, or eliminate data processing associated with individual action data structure operations. This saves processing power and other computing resources such as memory, reduces electrical power consumption by the data processing system and the reduced data transmissions via the computer network reduces bandwidth requirements and usage of the data processing system.
- The systems and methods described herein can include a data processing system that receives an input audio query, which can also be referred to as an input audio signal. From the input audio signal the data processing system can identify a request and a trigger keyword corresponding to the request. Based on the trigger keyword or the request, the data processing system can determine a first action data structure that includes at least one parameter defining a first action. The data processing system can also receive a second input audio signal, and from at least one request or trigger keyword of the second input audio signal, can identify from the second input audio signal a second action data structure that includes at least one parameter defining the second action. The data processing system can determine from the parameters of the first and second action data structures, at least one pooling parameter. The pooling parameter can indicate a level of overlap (e.g., subject matter commonality) between the first action data structure and the second action data structure.
- Based on the pooling parameter, the data processing system can combine the first action data structure and the second action data structure to generate a pooled data structure. The pooled data structure can indicate at least one parameter of actions (e.g., the first action or the second action) common to client computing devices that are the sources of the input audio signals. The data processing system can process the pooled data structure to generate output signals (e.g., an audio content item for rendering at one or more client computing devices), or to accomplish tasks related to the first and second actions. The data processing system can also transmit the pooled data structure to at least one service provider computing device, to cause the service provider to perform operations defined by the pooled data structure. These operations can correspond to the first or second actions, and can accomplish tasks related to the first or second actions. Pooling the action data structures causes the operations or tasks that correspond to the first or second actions to be performed or accomplished by the data processing system (or by the service provider computing device) with less data processing and less electrical power usage than would be the case without the pooling operation.
-
FIG. 1 depicts anexample system 100 to manage or otherwise optimize processing of voice activated threads in a voice activated data packet (or other protocol) based computer network environment. Thesystem 100 can include at least onedata processing system 105. Thedata processing system 105 can include at least one server having at least one processor. For example, thedata processing system 105 can include a plurality of servers located in at least one data center or server farm. Thedata processing system 105 can determine, from audio input signals, requests and trigger keywords associated with the request. Based on the requests and trigger keywords thedata processing system 105 can generate action data structures, each having respective parameters that indicate or define respective actions. Thedata processing system 105 can determine, from the respective parameters, a level of commonality or overlap between the action data structures. Responsive to a determination by the data processing system of a sufficient level of overlap, thedata processing system 105 can pool (or otherwise combine or merge) multiple different action data structures into a pooled action data structure. - The
data processing system 105 can include multiple, logically-grouped servers and facilitate distributed computing techniques. The logical group of servers may be referred to as a data center, server farm or a machine farm. The servers can be geographically dispersed. A data center or machine farm may be administered as a single entity, or the machine farm can include a plurality of machine farms. The servers within each machine farm can be heterogeneous—one or more of the servers or machines can operate according to one or more type of operating system platform. Thedata processing system 105 can include servers in a data center that are stored in one or more high-density rack systems, along with associated storage systems, located for example in an enterprise data center. Thedata processing system 105 with consolidated servers in this way can improve system manageability, data security, physical security of the system, and system performance by locating servers and high performance storage systems on localized high performance networks. Centralization of all or some of thedata processing system 105 components, including servers and storage systems, and coupling them with advanced system management tools allows more efficient use of server resources, which saves power and processing requirements and reduces bandwidth usage. - The
data processing system 105 can include at least one natural language processor (NLP)component 110, at least oneinterface 115, at least onepooling component 120, at least onecontent selector component 125, at least one audiosignal generator component 130, at least one direct action application programming interface (API) 135, at least onesession handler component 140, and at least onedata repository 145. TheNLP component 110,interface 115, poolingcomponent 120,content selector component 125, audiosignal generator component 130,direct action API 135, andsession handler component 140 can each include at least one processing unit, server, virtual server, circuit, engine, agent, appliance, or other logic device such as programmable logic arrays configured to communicate with thedata repository 145 and with other computing devices (e.g., theclient computing device 150, the content provider computing device 155, or the service provider computing device 160) via the at least onecomputer network 165. Thenetwork 165 can include computer networks such as the internet, local, wide, metro or other area networks, intranets, satellite networks, other computer networks such as voice or data mobile phone communication networks, and combinations thereof. - The
network 165 can include or constitute a display network, e.g., a subset of information resources available on the internet that are associated with a voice activated assistant system, content placement or search engine results system, or that are eligible to include third party content items as part of a content item placement campaign. Thenetwork 165 can be used by thedata processing system 105 to access information resources such as web pages, web sites, domain names, or uniform resource locators that can be presented, output, rendered, or displayed by theclient computing device 150. For example, via the network 165 a user of theclient computing device 150 can access information or data provided by the content provider computing device 155 or the serviceprovider computing device 160. - The
network 165 can include, for example a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communication network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, a SDH (Synchronous Digital Hierarchy) network, a wireless network or a wireline network, and combinations thereof. Thenetwork 165 can include a wireless link, such as an infrared channel or satellite band. The topology of thenetwork 165 may include a bus, star, or ring network topology. Thenetwork 165 can include mobile telephone networks using any protocol or protocols used to communicate among mobile devices, including advanced mobile phone protocol (“AMPS”), time division multiple access (“TDMA”), code-division multiple access (“CDMA”), global system for mobile communication (“GSM”), general packet radio services (“GPRS”) or universal mobile telecommunications system (“UMTS”). Different types of data may be transmitted via different protocols, or the same types of data may be transmitted via different protocols. - The
client computing device 150, the content provider computing device 155, and the serviceprovider computing device 160 can each include at least one logic device such as a computing device having a processor to communicate with each other or with thedata processing system 105 via thenetwork 165. Theclient computing device 150, the content provider computing device 155, and the serviceprovider computing device 160 can each include at least one server, processor or memory, or a plurality of computation resources or servers located in at least one data center. Theclient computing device 150, the content provider computing device 155, and the serviceprovider computing device 160 can each include at least one computing device such as a desktop computer, laptop, tablet, personal digital assistant, smartphone, portable computer, server, thin client computer, virtual server, or other computing device. - The
client computing device 150 can include at least onesensor 151, at least onetransducer 152, at least oneaudio driver 153, and at least onespeaker 154. Thesensor 151 can include a microphone or audio input sensor. Thetransducer 152 can convert the audio input into an electronic signal. Theaudio driver 153 can include a script or program executed by one or more processors of theclient computing device 150 to control thesensor 151, thetransducer 152 or theaudio driver 153, among other components of theclient computing device 150 to process audio input or provide audio output. Thespeaker 154 can transmit or render the audio output signal. - The
client computing device 150 can be associated with an end user that enters voice queries as audio input into the client computing device 150 (via the sensor 151) and receives audio output in the form of a computer generated voice that can be provided from the data processing system 105 (or the content provider computing device 155 or the service provider computing device 160) to theclient computing device 150, output from thespeaker 154. The computer generated voice can include recordings from a real person or computer generated language. - The content provider computing device 155 can provide audio based content items for display by the
client computing device 150 as an audio output content item. The content item can include an offer for a good or service, such as a voice based message that states: “Would you like me to order you a taxi?” For example, the content provider computing device 155 can include memory to store a series of audio content items that can be provided in response to a voice based query. The content provider computing device 155 can also provide audio based content items (or other content items) to thedata processing system 105 where they can be stored in thedata repository 145. Thedata processing system 105 can select the audio content items and provide (or instruct the content provider computing device 155 to provide) the audio content items to theclient computing device 150. The audio based content items can be exclusively audio or can be combined with text, image, or video data. - The
data processing system 105 can also perform conversions, e.g., accomplish tasks responsive to the input audio signal. For example, via thedirect action API 135, thedata processing system 105 can order a taxi responsive to a request to do so received from theclient computing device 150. Thedata processing system 105 can also perform a conversion by providing data (e.g., a pooled data structure) to the serviceprovider computing device 160 so that the service provider computing device can accomplish a task by, for example, ordering a taxi responsive to a request to do so received by thedata processing system 105 from theclient computing device 150. Thedata processing system 105 can also obtain conversion information, e.g., indicating that a task or action was accomplished, from theclient computing devices 150, from the content provider computing devices 155, or from the serviceprovider computing devices 160. - The service
provider computing device 160 can include at least one service provider natural language processor (NLP)component 161 and at least oneservice provider interface 162. The service provider NLP component 161 (or other components such as a direct action API of the service provider computing device 160) can engage with the client computing device 150 (via thedata processing system 105 or bypassing the data processing system 105) to create a back-and-forth real-time voice or audio based conversation (e.g., a session) between theclient computing device 150 and the serviceprovider computing device 160. For example, theservice provider interface 162 can receive or provide data messages to thedirect action API 135 of thedata processing system 105. The serviceprovider computing device 160 and the content provider computing device 155 can be associated with the same entity. For example, the content provider computing device 155 can create, store, or make available content items for a car sharing service, and the serviceprovider computing device 160 can establish a session with theclient computing device 150 to arrange for a delivery of a taxi or car of the car share service to pick up the end user of theclient computing device 150. Thedata processing system 105, via thedirect action API 135, theNLP component 110 or other components can also establish the session with the client computing device, including or bypassing the serviceprovider computing device 160, to arrange for example for a delivery of a taxi or car of the car share service. - The
data repository 145 can include one or more local or distributed databases, and can include a database management system. Thedata repository 145 can include computer data storage or memory and can store one ormore parameters 146, one ormore policies 147,content data 148, ortemplates 149 among other data. Theparameters 146,policies 147, andtemplates 149 can include information such as rules about a voice based session between theclient computing device 150 and the data processing system 105 (or the service provider computing device 160). Thecontent data 148 can include content items for audio output or associated metadata, as well as input audio messages that can be part of one or more communication sessions with theclient computing device 150. - The
system 100 can optimize or manage processing of voice activated threads (e.g., that indicate one or more actions) in a voice activated data packet (or other protocol) environment. For example, thedata processing system 105 can include or be part of a voice activated assistant service, voice command device, intelligent personal assistant, knowledge navigator, event planning, or other assistant program. Thedata processing system 105 can provide one or more instances of audio output for display from theclient computing device 150 to accomplish tasks related to this input audio signal. The tasks can include, for example, communicating with the serviceprovider computing device 160 or other third party computing devices to make dinner reservations or purchase movie tickets. For example, an end user can enter an input audio signal into theclient computing device 150 of: “OK, I would like to go to go dinner and then a movie tonight.” - The
data processing system 105 can include an application, script or program installed at theclient computing device 150, such as an app of a voice activated assistant system to communicate input audio signals to theinterface 115 of thedata processing system 105 and to drive components of the client computing device to render output audio signals. Thedata processing system 105 can receive data packets or other signal that includes or identifies an audio input signal. For example, thedata processing system 105 can execute or run theNLP component 110 to receive the audio input signal. - The audio input signal can be detected by the sensor 151 (e.g., a microphone) of the client computing device. Via the
transducer 152, theaudio driver 153, or other components theclient computing device 150 can provide the audio input signal to the data processing system 105 (e.g., via the network 165) where it can be received (e.g., by the interface 115) and provided to theNLP component 110 or stored in thedata repository 145 ascontent data 148. - The
NLP component 110 can obtain the input audio signal. From the input audio signal, theNLP component 110 can identify at least one request or at least one trigger keyword corresponding to the request. The request can indicate intent or subject matter of the input audio signal. The trigger keyword can indicate a type of action likely to be taken. For example, theNLP component 110 can parse the input audio signal to identify at least one request to leave home for the evening to attend dinner and a movie. The trigger keyword can include at least one word, phrase, root or partial word, or derivative indicating an action to be taken. For example, the trigger keyword “go” or “to go to” from the input audio signal can indicate a need for transport. In this example, the input audio signal (or the identified request) does not directly express an intent for transport, however the trigger keyword indicates that transport is an ancillary or sub-action to at least one other action that is indicated by the request. - The
NLP component 110 can convert the audio input signal into recognized text by comparing the input signal against a stored, representative set of audio waveforms (e.g., in the data repository 145) and choosing the closest matches. The representative waveforms are generated across a large set of users, and can be augmented with speech samples. After the audio signal is converted into recognized text, theNLP component 110 can match the text to words that are associated, for example via training across users or through manual specification, with actions that thedata processing system 105 can serve. - The
direct action API 135 can execute programs or scripts, for example from theNLP component 110, thepooling component 120, or thecontent selector component 125 to identify content items for one or more of these actions. Thedirect action API 135 can execute a specified action to satisfy the end user's intention, as determined by thedata processing system 105. Depending on the action specified in its inputs, thedirect action API 135 can execute code or a dialog script that identifies the parameters required to fulfill a user request. Such code can lookup additional information, e.g., in thedata repository 145, such as the name of a home automation service, or it can provide audio output for rendering at theclient computing device 150 to ask the end user questions such as the intended destination of a requested taxi, for example. Thedirect action API 135 can determine necessary parameters and can package the information into an action data structure, which can then be sent to another component such as thecontent selector component 125 or to the serviceprovider computing device 160 to be fulfilled. Thedirection action API 135 can execute on, or include, hardware of thedata processing system 105 such as one or more processors of one or more servers in one or more data centers. - The
direct action API 135 of thedata processing system 105 can generate, based on the request or the trigger keyword, data structures for the actions of the thread. The action data structures can be generated responsive to the request. The action data structure can be included in the messages that are transmitted to or received by the serviceprovider computing device 160, e.g. as part of or associated with pooled data structures. Based on the request parsed by theNLP component 110, thedirect action API 135 can determine to which of a plurality of serviceprovider computing devices 160 the message should be sent. For example, if an input audio signal includes “order a taxi,” theNLP component 110 can identify the trigger word “order” and the request for a taxi. Thedirect action API 135 can package the request into an action data structure for transmission as a message to a serviceprovider computing device 160 of a taxi service. The message can also be passed to thecontent selector component 125. The action data structure can include information for completing the request. In this example, the information can include a pick up location and a destination location. Thedirect action API 135 can retrieve atemplate 149 from therepository 145 to determine which fields to include in the action data structure. Thedirect action API 135 can retrieve content from therepository 145 to obtain information for the fields of the data structure. Thedirect action API 135 can populate the fields from the template with that information to generate the data structure. Thedirect action API 135 can also populate the fields with data from the input audio signal. Thetemplates 149 can be standardized for categories of service providers or can be standardized for specific service providers. For example, ride sharing service providers can use the followingstandardized template 149 to create the data structure: {client_device_identifier; authentication_credentials; pick_up_location; destination_location; no_passengers; service_level}. - The
NLP component 110 can obtain the input audio signal as data packet or other protocol based transmissions from theclient computing device 150. For example, the input audio signal could be a voice input into the sensor 152 (e.g., a microphone) of “OK, take me home”. From this input audio signal theNLP component 110 can identify at least one request or at least one trigger keyword. The request can indicate subject matter of the input audio signal, e.g., travel. The trigger keyword can indicate an action, e.g., “take” or “home” can indicate a need for a car share service to a destination associated with a home destination of theclient computing device 150 that originated the input audio signal. - The
direct action API 135 can obtain the requests or trigger keywords that correspond to the input audio signal, and from this information can generate an action data structure. The action data structure can include information used, for example, to communicate with the service provider computing device to reserve a car from a car share service to take the end user (who originated the input audio signal into the client computing device) to a destination. The action data structure can, with end user consent, include parameters such as information identifying the location of theclient computing device 150 to arrange the pickup by the car share service, as well as information identifying the destination location. The parameters can be stored in and retrieved from the data repository (e.g., parameters 146) bydata processing system 105 components. Thedirection action API 135 can also generate the action data structure to include or indicate parameters such as account information of an account associated with the car share service. For example, thedirect action API 135 can obtain, with end user consent, this account information (or other parameter) from an app or program for the car share service that is installed on theclient computing device 150. The parameters can indicate or define actions associated with the action data structures. Thedirect action API 135 can be installed on thedata processing system 105, remote from theclient computing device 150. Thedirect action API 135 can perform a direct conversion, e.g., place an order with the serviceprovider computing device 160 for the car share service trip, or indirectly by providing the action data structure to the serviceprovider computing device 160 for the serviceprovider computing device 160 to process the action data structure in order to complete the car reservation transaction. - The
NLP component 110 can obtain multiple input audio signals from the same or differentclient computing devices 150. For each input audio signal, the NLP component can identify or create corresponding requests or corresponding trigger keywords. Thedirect action API 135 can generate, based on the respective requests or keywords, action data structures corresponding to the respective input audio signals. For example, theNLP component 110 can receive a first input audio signal indicating the voice input “OK, take me home”. In response, thedirect action API 135 can generate a first action data structure that can be used to reserve a car from a car share service for the end user associated with the first input audio signal. TheNLP component 110 can also receive a second input audio signal indicating, for example, “OK, can I get a taxi please”. In response, thedirect action API 135 can generate a second action data structure for a car share service for the end user associated with the second input audio signal. - The first and second input audio signal can originate from the same
client computing device 150 or from differentclient computing devices 150, and can be associated with the same or different end users. For example, theNLP component 110 can receive first input audio signal from a firstclient computing device 150 and can receive a second input audio signal from a secondclient computing device 150 that can be associated with different end users. From this information, which can include metadata or header information of packet based transmissions, among supplemental information that can be obtained from sources other than the input audio signal, such as fromcontent data 148 of thedata repository 145, thedirect action API 135 can generate a first data structure for the first input audio signal and a second data structure for the second input audio signal. The first and second data structures can both indicate requests to reserve a car from a car share service. Each action data structure can include parameters that indicate their own origination and destination locations, or can indicate different accounts that different end users have with the car share service (or other parameters). The action data structure can also include one or more parameters that indicate that a client computing device 150 (or associated end user) does not have an account with the car share service or other entity. - Based for example on parameters of multiple action data structures, the
pooling component 120 can identify overlap or similarities between two or more different action data structures that were generated from different input audio signals. For example, one or more processors of thedata processing system 105 can include or execute thepooling component 120, remote from theclient computing device 105 and remote from the serviceprovider computing device 160, to identify overlapping or common subject matter of two or more different action data structures. Thepooling component 120 can apply a heuristic technique to identify with a threshold degree of certainty of common patterns among the parameters of the different action data structures. - The level of overlap can be or include a similarity metric that indicates a common feature of multiple action data structures. The
pooling component 120 can determine a similarity metric indicating the multiple action data structures relate to time data, location data, service provider entity data, subject matter data, or geographic data. For example, thepooling component 120 can determine that multiple action data structures include requests for transport from the same transportation company, or reservations for different times or evenings for the same restaurant, requests to travel to or from a common destination or geographic area, or requests to order one or more products from one entity. Thepooling component 120 can identify the similarity metrics or common features as at least one pooling parameter, and based on the pooling parameter the pooling component can create the batched or pooled data structure. The overlap may but need not include at least partially identical code or partially identical action data structures. For example, thepooling component 120 can identify overlap based on requests for services from the same entity, or based on commonality among parameters of the action data structures such as a similar, but not identical, time or location data. - For example, parameters of the first action data structure can indicate a request for a car share service from a pickup location of ‘point A’ to a destination of ‘point B’. Parameters of the second action data structure can indicate a different request for a car share service from a pickup location of ‘point C’ to a destination of ‘point D’. By applying a heuristic or other statistical technique, the
data processing system 105 or component thereof (e.g., the pooling component 120) can determine that pickup location ‘point C’ is located between, or within a threshold distance (e.g., 1 mile) of a route between, ‘point A’ and ‘point B’. Thepooling component 120 can also determine that ‘point D’ is located within a threshold distance (e.g., 5 miles) of destination ‘point B’. While the parameters of the different action data structures (e.g., parameters indicating different points A-D, or other information such as different accounts associated with differentclient computing devices 150, thepooling component 120 can determine that sufficient overlap or commonality exists between the parameters due, for example to route similarities satisfying threshold metrics. Thepooling component 120 can also determine that the two action data structures overlap because their respective parameters indicate that they are associated with respective input audio signals that were made within a threshold time period of one another, such as within 1 minute, one hour, of other time period of each other, or because their respective parameters indicate that they are both requesting a good or service from the same entity. - The
pooling component 120 can generate, based on the analysis of at least two action data structures, at least one pooling parameter. The pooling parameter can indicate common subject matter, features, entities, or parameters from among two or more action data structures. For example, the pooling parameter can indicate that the requests for car share services indicated by parameters of the first and second action data structures share similar locations, or similar routes between respective sources and destinations. The direction action API 135 (or the service provider computing device 160) can use this information to service the actions associated with or indicated by the action data structures, such as car service rides between respective sources and destinations. - Referring to
FIG. 1 andFIG. 2 , among others, based on the determined level of overlap, thepooling component 120 can create at least one pooleddata structure 200. The pooled data structure can include a packet based architecture (or other format) and can include aheader 205. Theheader 205 can include supplemental data identifying thedata processing system 105, source or destination address information, governing protocol information, IP addresses, frame synchronization data, size data, resolution data, or metadata associated with action data structures. The payload or body of the pooleddata structure 200 can include at least onepooling parameter 210, or other information such as at least onefirst parameter 215 or at least onesecond parameters 220. Thefirst parameter 215 can include a parameter of the firstaction data structure 225, and thesecond parameters 220 can include a parameter of the secondaction data structure 230. Thefirst parameter 215,second parameter 220, firstaction data structure 225 and the secondaction data structure 230 may but need not be included in the pooleddata structure 200. For example, the body of the pooleddata structure 200 can include only the pooling parameter 210 (and not include any of thefirst parameter 215, thesecond parameter 220, the firstaction data structure 225 or the second action data structure 230), or the body of the pooleddata structure 200 can include the poolingparameter 210 as well as one or more of thefirst parameter 215, thesecond parameter 220, the firstaction data structure 225 or the secondaction data structure 230. The pooleddata structure 200 can include one or more individual packets transmitted separately in sequence or parallel as part of one or more data transmissions between thedata processing system 105 and the serviceprovider computing device 160. The pooling parameter, or information inheaders 205 can indicate that the separate transmissions or separate data packets are associated with the same batch transmission, e.g., the same overall pooleddata structure 200. - Referring to
FIG. 3 , among others, thedata processing system 105 can receive, from a firstclient computing device 150, the firstinput audio signal 305, (e.g., “OK, take me home”). Thedata processing system 105 can also receive, from a secondclient computing device 150, the secondinput audio signal 310, (e.g., “OK, can I get a taxi please”). Thedata processing system 105 can determine respective parameters that define actions associated with the firstinput audio signal 305 and the secondaudio input signal 305.Data processing system 105 components such as thepooling component 120 can apply statistical or heuristic techniques to the parameters associated with the firstinput audio signal 305 and the secondinput audio signal 310 to generate at least one pooling parameter indicative of subject matter overlap or commonality between the firstinput audio signal 305 and the secondinput audio signal 310. For example, the pooling parameter can indicate that the firstinput audio signal 305 and the second input audio signal both relate to requests to hire a car from a car share service. Thedata processing system 105 can batch or pool the action data structures corresponding to the firstinput audio signal 305 and the secondinput audio signal 310 to create the batched or pooleddata structure 200. Thedata processing system 105 can transmit or otherwise provide the pooleddata structure 200 to the serviceprovider computing device 160 for processing by the serviceprovider computing device 160 to reserve at least one car responsive to the first and second input audio signals 305, 310. - The pooled
data structure 200 can include pooling parameters generated from more than two action data structures (or more than two input audio signals). For example, a third audio input signal can cause thedata processing system 105 to generate a third action data structure for a third action. Thepooling component 120 can identify at least one pooling parameter based on parameters of the third action data structure. Responsive to this determination, thepooling component 120 can modify the pooleddata structure 200 to include or indicate an association with the third data structure, or can include a new pooled data structure associated with three (or more) input audio signals. - By consolidating individual action data structures into at least one pooled
data structure 200 and transmitting the pooled data structure(s) 200 to the serviceprovider computing device 160, rather than the individual action data structures for eachinput audio signal data processing system 105 can reduce the number, size, or frequency of data transmissions via thenetwork 165. This can reduce bandwidth utilization of network transmissions, and can reduce processing power and memory storage requirements of both the data processing system 105 (e.g., data repository 145) and of the serviceprovider computing device 160 that would otherwise be used to process and transmit individual action data structures that may have a level of redundancy or subject matter overlap contained within their data. - The pooled
data structure 200 may but need not be a single data structure. For example the pooled data structure can include multiple individual data structures transmitted by thedata processing system 105, with or without the pooling parameters, as part of a batched or periodic transmission to one or more serviceprovider computing devices 160 controlled by one entity, such as a car share service provider. - The
data processing system 105 and the serviceprovider computing device 160 can correspond to different entities. For example, the data processing system 105 (and components such as the direct action API 135) can be part of a voice based assistant computer system designed to interface with end users of theclient computing device 150 to provide audio responses to audio input signals. For example, the audiosignal generator component 130 can create an audio response of “There is a taxi 5 minutes away, would you like to order it?” to the audio input signal of “OK, take me home”. Thesession handler component 140 can establish a communication session via thecomputer network 165 between thedata processing system 105 and theclient computing device 150 to provide this audio response. The audio response can cause theaudio driver 153 of theclient computing device 150 to cause thespeaker 154 to output acoustic waves to render the audio response to the end user. - Rather than being part of the voice based assistant computer system, the service
provider computing device 160 can be a computing device of the car share or taxi service entity. Thedata processing system 105, and not the serviceprovider computing device 160, processes the input audio signals 305, 310 to generate the pooleddata structure 200. Thus, in this example the audio input signals 305, 310 correspond to actions for the serviceprovider computing device 160, e.g., ordering a car from the car share service entity that controls the serviceprovider computing device 160. Thedata processing system 105 can operate as an intermediary appliance or device that can hook, intercept, or otherwise receive the input audio signals that include requests for action by the serviceprovider computing device 160. Thedata processing system 105 can batch or consolidate these requests into at least one pooleddata structure 200 that is provided to the serviceprovider computing device 160. In this example, rather than receiving input audio signals directly from the client computing devices 150 (e.g., bypassing the data processing system 105), the serviceprovider computing device 160 instead receives a pooled data structure 200 (e.g., a batched transmission of consolidated action data structures) from the intermediarydata processing system 105. In this example, requests from theclient computing devices 150 for goods or services of the serviceprovider computing device 160 are routed through thedata processing system 105, where they are consolidated into a pooleddata structure 200 that can be transmitted to the serviceprovider computing device 160. - Responsive to receipt of the pooled
data structure 200, the serviceprovider computing device 160 can perform at least one operation defined or indicated by the pooleddata structure 200. The operation can correspond to one or more of the respective actions of the action data structures of the first or second input audio signals 305, 310. For example, the operations can include reserving a car from a car share service. The serviceprovider computing device 160 can provide responses to theclient computing devices 150. For example, the serviceprovider computing device 160 can provide afirst response 315 to the firstclient computing device 150, and can provide asecond response 320 to the secondclient computing device 150. The first andsecond responses data processing system 105 during transmission from the serviceprovider computing device 160 to the first and secondclient computing devices 150. Theresponses 315 can include text, image, or other data messages (including audio) that confirm a conversion action, such as a reservation for a car from a car service. Thedata processing system 105 can receive an indication of the conversion (or of theresponses 315, 320) via thecomputer network 165 from the serviceprovider computing device 160 or from theclient computing devices 150. Thedata processing system 105 can also receive theresponses - The service provider entity that corresponds to one or more service
provider computing devices 160 can perform multiple operations to accomplish different actions indicated by the pooleddata structure 200. For example, the service provider entity can provide grocery deliveries and operate a car share service for human transport. The pooleddata structure 200 can indicate both of these actions, which can be accomplished by a single service provider entity that controls one or more serviceprovider computing devices 160. The serviceprovider computing devices 160 can also provide pooling policy data to thedata processing system 105. For example thedirect action API 135 can receive from the serviceprovider computing devices 160 pooling policy data. The pooling policy data can provide policy guidelines used by thepooling component 120 to generate the pooleddata structure 200. The pooling policy data can be stored in the data repository 145 (e.g., as policies 147). The pooling policy data can indicate a format of the pooled data structure, template information (e.g., template 149) or information size data, or about when, how, how frequently, or what information should be included in the pooled data structure. Thepooling component 120 can create the pooleddata structure 200 in compliance with the pooling policy data. - In some examples, the
responses client computing devices 150. For example, the serviceprovider computing device 160 can consolidate the first and second actions, indicated by respective action data structures in the pooleddata structure 200 and can provide a single car the arrives in sequence at pickup locations associated with the first and secondclient computing devices 150. - The
responses responses action data structure 225 or the second action data structure 230 (or the first andsecond parameters 215, 220) of the pooleddata structure 200. For example, the serviceprovider computing devices 160 can provide theresponse 315 with information about a first car share reservation for the firstclient computing device 150, and can provide theresponse 320 with information about a different, second car share reservation for the secondclient computing device 150. - The pooling parameter created by the
pooling component 120 and used to generate the pooleddata structure 200 can also include temporal, destination, or location data. For example, the pooling parameter can indicate a plurality of action data structures that each correspond to requests for services from a single entity associated with the service provider computing device (e.g., destination information), such as reservation requests for a particular restaurant, orders for a good manufactured by the entity, or car share reservation requests from a car share service. The pooling parameter can also indicate a number of action data structures made within a defined time period, such as within a 2 minute interval each requesting a car service from the same car share entity. The pooling parameter can include location information, such as location information identifying a defined geographical area (e.g., a city block or neighborhood) from which a plurality of people provide input audio signals requesting a service from the same entity associated with the serviceprovider computing device 160. From the pooling parameter, thepooling component 120 can generate the pooleddata structure 200 and provide it to the serviceprovider computing device 160, which can perform operations responsive to the pooled data structure, and provide associatedresponses client computing devices 150. - The pooled
data structure 200 can include only operations for a single entity, which is a different entity than the entity associated with thedata processing system 105. For example, the data processing system 105 (or component such as the pooling component 120) can provide a first pooled data structure to one or more first serviceprovider computing devices 160 controlled by a first entity, and can generate and provide a second pooled data structure to one or more second serviceprovider computing devices 160 that are controlled by a different, second entity unrelated to the first entity. For example, the first pooled data structure can relate to consolidated requests to buy tickets for a local sporting event from a ticket broker company, and the second pooled data structure can relate to requests to make travel reservations for an overseas tropical beach vacation from a travel agent company. - The
content selector component 125 can obtain indications of at least one action of the action data structures of the pooledaction data structure 200. From the information received by thecontent selector component 125, e.g., an indication of an action, thecontent selector component 125 can identify at least one content item. The content item can be responsive or related to the third action. For example, the content item can include an audio message offering services of a car share company, responsive to the third action that indicates a transportation need. Thecontent selector component 125 can query thedata repository 145 to select or otherwise identify the content item, e.g., from thecontent data 148. Thecontent selector component 125 can also select the content item from the content provider computing device 155. For example responsive to a query received from thedata processing system 105, the content provider computing device 155 can provide a content item to the data processing system 105 (or component thereof) for eventual output by theclient computing device 150. The content item can relate to a service provider associated with the action data structure, e.g., an identified car share service, or another service such as a competitor or ancillary service. - The audio
signal generator component 130 can generate or otherwise obtain an output signal that includes the content item. For example, thedata processing system 105 can execute the audio signal generator component to generate or create an output signal corresponding to the content item. Theinterface 115 of thedata processing system 105 can provide transmit one or more data packets that include the output signal via thecomputer network 165 to theclient computing device 150. Theinterface 115 can be designed, configured, constructed, or operational to receive and transmit information using, for example, data packets. Theinterface 115 can receive and transmit information using one or more protocols, such as a network protocol. Theinterface 115 can include a hardware interface, software interface, wired interface, or wireless interface. Theinterface 115 can facilitate translating or formatting data from one format to another format. For example, theinterface 115 can include an application programming interface that includes definitions for communicating between various components, such as software components of thesystem 100. - The
data processing system 105 can provide the output signal (e.g., corresponding to the content item or responsive to the input audio signal), from thedata repository 145 or from the audiosignal generator component 130 to theclient computing device 150. Thedata processing system 105 can also instruct, via data packet transmissions, the content provider computing device 155 or the serviceprovider computing device 160 to provide the output signal to theclient computing device 150. The output signal can be obtained, generated, transformed to or transmitted as one or more data packets (or other communications protocol) from the data processing system 105 (or other computing device) to theclient computing device 150. - The
content selector component 125 can select the content item as part of a real-time content selection process. For example, the content item can be provided to theclient computing device 150 as audio output in a conversational manner in direct response to the input audio signal. The real-time content selection process to identify the content item and provide the content item to theclient computing device 150 can occur within one minute or less from the time of the input audio signal and be considered real-time. - The output signal that corresponds to the content item, for example obtained or generated by the audio
signal generator component 130 transmitted via theinterface 115 and thecomputer network 165 to theclient computing device 150, can cause theclient computing device 150 to execute theaudio driver 153 to drive thespeaker 154 to generate an acoustic wave corresponding to the output signal. The acoustic wave can include words of or corresponding to the content item for the third action. The acoustic wave can be output from theclient computing device 150, e.g., as an acoustic wave that include the audio output of “Your taxi is confirmed” or “While you are waiting for your taxi, there is a coffee shop around the corner”. - The
data processing system 105 can prevent or delay transmission of the pooled data structure to the serviceprovider computing device 160. For example, theinterface 115 or a script executing via thedirect action API 135 or thepooling component 120 can prevent transmission of data packets that include the pooled data structure until a triggering event. The triggering event can include expiration of a pre-determined time period, from less than one minute to a period of days. The triggering event can also be based on available bandwidth, processing, or memory. For example, the serviceprovider computing device 160 can prompt or query the data processing system for the pooled data structure, which thedata processing system 105 transmits in response to the prompt. - The
data processing system 105 can also delay transmission of the pooled data structure to optimize processing utilization. For example, thedata processing system 105 can delay content item transmission until an off-peak or non-peak period of data center usage, which results in more efficient utilization of the data center by reducing peak bandwidth usage, heat output or cooling requirements. Thedata processing system 105 can also initiate a conversion or other activity associated with the content item, such as ordering a car service, based on data center utilization rates or bandwidth metrics or requirements of thenetwork 165 or of a data center that includes thedata processing system 105. This can reduce network traffic, load requirements, and save processing and electrical power. - The
data processing system 105 can receive a response to a content item such as an offer for a ride share service that states “would you like a ride home from the movie theater?” The response can include an input audio signal, e.g., “yes please” that was entered into theclient computing device 150 by the end user and received by thedata processing system 105. Based on this response, thedirect action API 135 can communicate with the service provider computing device 160 (that can be associated with the content item, such as a car share company) to order a taxi or ride share vehicle for the location of the movie theater at the time the movie ends. Thedata processing system 105 can obtain this location or time information as part of the data packet (or other protocol) based data message communication with theclient computing device 150, from thedata repository 145, or from other sources such as the serviceprovider computing device 160 or the content provider computing device 155. Confirmation of this order (or other conversion) can be provided as an audio communication from thedata processing system 105 to theclient computing device 150 in the form of an output signal from thedata processing system 105 that drives theclient computing device 150 to render audio output such as, “great, you will have a car waiting for you at 11 pm outside the theater.” Thedata processing system 105, via thedirect action API 135, can communicate with the serviceprovider computing device 160 to confirm the order for the car. - The
data processing system 105 can obtain the response (e.g., “yes please”) to the content item (“would you like a ride home from the movie theater?”) and can route a packet based data message to the service provider NPL component 161 (or other component of the service provider computing device). This packet based data message can cause the serviceprovider computing device 160 to effect a conversion, e.g., to make a car pick up reservation outside the movie theater. Thedata processing system 105 can obtain a notification of the conversion, e.g., from the serviceprovider computing device 160, the content provider computing device 155, or theclient computing device 150. - Based on a response to a content item for a subsequent action in the thread, such as the response “yes please” to the content item “would you like a ride home from the movie theater”, or based upon the action data structures including the pooled data structure that can directly indicate a request for a good or service, the
data processing system 105 can initiate a conversion or action. For example, processors of thedata processing system 105 can invoke thedirect action API 135 to execute scripts that order a car from a car share service. Thedirect action API 135 can obtain content data 148 (orparameters 146 or policies 147) from thedata repository 145, as well as data received with end user consent from theclient computing device 150 to determine location, time, user accounts, logistical or other information in order to reserve a car from the car share service. Using thedirect action API 135, thedata processing system 105 can also communicate with the serviceprovider computing device 160 to complete the conversion by in this example making the car share pick up reservation. Thedirect action API 135 can initiate the conversion or activity to accomplish an action associated with the pooled data structure (e.g., an action defined by one or more parameters of an action data structure indicated by the pooled data structure) -
FIG. 4 depictsmethod 400 to manage voice activated threads in a voice activated data packet based computer network environment. Themethod 400 can receive at least one first audio input signal (ACT 405). For example, via theinterface 115 the NLP component can receive data packets (ACT 405). Thedata processing system 105 can execute, launch, or invoke theNLP component 110 to receive packet or other protocol based transmissions via the network from a firstclient computing device 150. The data packets can include or correspond to an input audio signal detected by thesensor 151 of a firstclient computing device 150, such as a first end user saying “OK, take me home” into theclient computing device 150, e.g., smartphone. Themethod 400 can parse the input audio signal to identify at least one first request or at least one first trigger keyword from the input audio signal (ACT 410). For example, theNLP component 110 can parse the input audio signal to identify requests (to go “home”) as well as trigger keywords (“take”) that correspond or relate to the request. Themethod 400 can generate at least one first action data structure (ACT 415) based on the identified requests or keywords associated with the received first audio input signal. For example, thedirect action API 135 can generate an action data structure that includes at least one parameter defining a first action, such as a location for a taxi pick-up, a requested service provider, or time information. - The
method 400 can receive at least one second audio input signal (ACT 420). For example, theNLP component 110 can receive packet or other protocol based transmissions via the network from a secondclient computing device 150. The data packets can include or correspond to an input audio signal detected by thesensor 151 of the secondclient computing device 150, such as an end user saying “OK, can I get a taxi please” into the secondclient computing device 150. Themethod 400 can identify at least one second request or at least one second trigger keyword from the second input audio signal (ACT 425). For example, theNLP component 110 can parse the input audio signal to identify requests (to go “taxi”) as well as trigger keywords (“get”) that correspond or relate to the request. Themethod 400 can generate at least one second action data structure (ACT 430) based on the identified requests or keywords associated with the received first audio input signal. For example, thedirect action API 135 can generate an action data structure that includes at least one parameter defining a second action, such as a location for a taxi pick-up, a requested service provider, or time information. - The
method 400 can determine at least one pooling parameter (ACT 435). For example, based on parameters of respective actions that indicate a common requested service provider or other commonality, (e.g., a level of overlap between action data structures), thepooling component 120 can determine a pooling parameter that indicates the common requested service provider. Themethod 400 can generate a pooled data structure (ACT 440). For example, based on the indication of the common requested service provider thepooling component 120 can generate at least one pooled data structure that indicates the first and second action data structures, or their associated actions or parameters (ACT 440). Themethod 400 can transmit the pooled data structure (ACT 445). For example, thedata processing system 105 can provide the pooled data structure to one or more serviceprovider computing devices 160 controlled by the common requested service provider entity. -
FIG. 5 is a block diagram of anexample computer system 500. The computer system orcomputing device 500 can include or be used to implement thesystem 100, or its components such as thedata processing system 105. Thecomputing system 500 includes a bus 505 or other communication component for communicating information and aprocessor 510 or processing circuit coupled to the bus 505 for processing information. Thecomputing system 500 can also include one ormore processors 510 or processing circuits coupled to the bus for processing information. Thecomputing system 500 also includesmain memory 515, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 505 for storing information, and instructions to be executed by theprocessor 510. Themain memory 515 can be or include thedata repository 145. Themain memory 515 can also be used for storing position information, temporary variables, or other intermediate information during execution of instructions by theprocessor 510. Thecomputing system 500 may further include a read only memory (ROM) 520 or other static storage device coupled to the bus 505 for storing static information and instructions for theprocessor 510. Astorage device 525, such as a solid state device, magnetic disk or optical disk, can be coupled to the bus 505 to persistently store information and instructions. Thestorage device 525 can include or be part of thedata repository 145. - The
computing system 500 may be coupled via the bus 505 to adisplay 535, such as a liquid crystal display, or active matrix display, for displaying information to a user. An input device 530, such as a keyboard including alphanumeric and other keys, may be coupled to the bus 505 for communicating information and command selections to theprocessor 510. The input device 530 can include atouch screen display 535. The input device 530 can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to theprocessor 510 and for controlling cursor movement on thedisplay 535. Thedisplay 535 can be part of thedata processing system 105, theclient computing device 150 or other component ofFIG. 1 , for example. - The processes, systems and methods described herein can be implemented by the
computing system 500 in response to theprocessor 510 executing an arrangement of instructions contained inmain memory 515. Such instructions can be read intomain memory 515 from another computer-readable medium, such as thestorage device 525. Execution of the arrangement of instructions contained inmain memory 515 causes thecomputing system 500 to perform the illustrative processes described herein. One or more processors in a multi-processing arrangement may also be employed to execute the instructions contained inmain memory 515. Hard-wired circuitry can be used in place of or in combination with software instructions together with the systems and methods described herein. Systems and methods described herein are not limited to any specific combination of hardware circuitry and software. - Although an example computing system has been described in
FIG. 5 , the subject matter including the operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. - For situations in which the systems discussed herein collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features that may collect personal information (e.g., information about a user's social network, social actions or activities, a user's preferences, or a user's location), or to control whether or how to receive content from a content server or other data processing system that may be more relevant to the user. In addition, certain data may be anonymized in one or more ways before it is stored or used, so that personally identifiable information is removed when generating parameters. For example, a user's identity may be anonymized so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, postal code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about him or her and used by the content server.
- The subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more circuits of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, data processing apparatuses. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. While a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.
- The terms “data processing system” “computing device” “component” or “data processing apparatus” encompass various apparatuses, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures. The
direct action API 135,content selector component 125, poolingcomponent 120 orNLP component 110 and otherdata processing system 105 components can include or share one or more data processing apparatuses, systems, computing devices, or processors. - A computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
- The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs (e.g., components of the data processing system 105) to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatuses can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
- The subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or a combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).
- The computing system such as
system 100 orsystem 500 can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network (e.g., the network 165). The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server transmits data (e.g., data packets representing a content item) to a client computing device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client computing device). Data generated at the client computing device (e.g., a result of the user interaction) can be received from the client computing device at the server (e.g., received by thedata processing system 105 from thecomputing device 150 or the content provider computing device 155 or the service provider computing device 160). - While operations are depicted in the drawings in a particular order, such operations are not required to be performed in the particular order shown or in sequential order, and all illustrated operations are not required to be performed. Actions described herein can be performed in a different order.
- The separation of various system components does not require separation in all implementations, and the described program components can be included in a single hardware or software product. For example, the
NLP component 110, thecontent selector component 125, or thepooling component 120 can be a single component, app, or program, or a logic device having one or more processing circuits, or part of one or more servers of thedata processing system 105. - Having now described some illustrative implementations, it is apparent that the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to accomplish the same objectives. Acts, elements and features discussed in connection with one implementation are not intended to be excluded from a similar role in other implementations or implementations.
- The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including” “comprising” “having” “containing” “involving” “characterized by” “characterized in that” and variations thereof herein, is meant to encompass the items listed thereafter, equivalents thereof, and additional items, as well as alternate implementations consisting of the items listed thereafter exclusively. In one implementation, the systems and methods described herein consist of one, each combination of more than one, or all of the described elements, acts, or components.
- Any references to implementations or elements or acts of the systems and methods herein referred to in the singular may also embrace implementations including a plurality of these elements, and any references in plural to any implementation or element or act herein may also embrace implementations including only a single element. References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their components, acts, or elements to single or plural configurations. References to any act or element being based on any information, act or element may include implementations where the act or element is based at least in part on any information, act, or element.
- Any implementation disclosed herein may be combined with any other implementation or embodiment, and references to “an implementation,” “some implementations,” “one implementation” or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be included in at least one implementation or embodiment. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.
- References to “or” may be construed as inclusive so that any terms described using “or” may indicate any of a single, more than one, and all of the described terms. For example, a reference to “at least one of ‘A’ and ‘B’” can include only ‘A’, only ‘B’, as well as both ‘A’ and ‘B’. Such references used in conjunction with “comprising” or other open terminology can include additional items.
- Where technical features in the drawings, detailed description or any claim are followed by reference signs, the reference signs have been included to increase the intelligibility of the drawings, detailed description, and claims. Accordingly, neither the reference signs nor their absence have any limiting effect on the scope of any claim elements.
- The systems and methods described herein may be embodied in other specific forms without departing from the characteristics thereof. For example, receipt by the
data processing system 105 of multiple audio input signals (or identification or respective requests or trigger keywords) can occur simultaneously or in sequence from one or more sources. The foregoing implementations are illustrative rather than limiting of the described systems and methods. Scope of the systems and methods described herein is thus indicated by the appended claims, rather than the foregoing description, and changes that come within the meaning and range of equivalency of the claims are embraced therein.
Claims (20)
1. A system to manage voice activated threads in a voice activated data packet based computer network environment, comprising:
a natural language processor component executed by a data processing system to receive, via an interface of the data processing system, data packets comprising a first input audio signal detected by a sensor of a first client computing device;
the natural language processor component to parse the first input audio signal to identify a first request and a first trigger keyword corresponding to the first request;
a direct action application programming interface (“API”) of the data processing system to generate, based on the first trigger keyword and in response to the first request, a first action data structure with a parameter defining a first action;
the natural language processor component to receive, via the interface of the data processing system, data packets comprising a second input audio signal detected by a sensor of a second client computing device, and to parse the second input audio signal to identify a second request and a second trigger keyword corresponding to the second request;
the direct action API to generate, based on the second trigger keyword and in response to the second request, a second action data structure with a parameter defining a second action; and
a pooling component of the data processing system to:
determine, based on a heuristic technique applied to the parameter of the first action data structure and the parameter of the second action data structure, a pooling parameter that indicates a level of overlap between the first action data structure and the second action data structure, the pooling parameter used to generate a pooled data structure that causes a reduction in processor or bandwidth utilization as compared to separate transmissions of the first action data structure and the second action data structure;
generate, based on the pooling parameter, the first action data structure, and the second action data structure, the pooled data structure; and
transmit, via a computer network, the pooled data structure to a service provider computing device to cause the service provider computing device to perform an operation defined by the pooled data structure and corresponding to the first action and the second action.
2. The system of claim 1 , wherein the service provider computing device is a first service provider computing device corresponding to a first entity, comprising:
the pooling component of the data processing system to transmit, via the computer network, a second pooled data structure to a second service provider computing device corresponding to a second entity to cause the second service provider computing device to perform an operation defined by the second pooled data structure.
3. The system of claim 1 , wherein the pooling parameter is a first pooling parameter, the pooled data structure is a first pooled data structure, and the service provider computing device is a first service provider computing device, comprising the pooling component of the data processing system to:
determine a second pooling parameter that indicates a level of overlap between a third action data structure and a fourth action data structure;
generate, based on the second pooling parameter, the third action data structure, and the fourth action data structure, a second pooled data structure; and
transmit, via the computer network, the second pooled data structure to a second service provider computing device to cause the second service provider computing device to perform an operation defined by the second pooled data structure and corresponding to the third action data structure and the fourth action data structure.
4. The system of claim 1 , wherein the pooling parameter is a first pooling parameter, comprising:
the natural language processor component to receive, via the interface of the data processing system, data packets comprising a third input audio signal detected by the sensor of the first client computing device, and to parse the third input audio signal to identify a third request and a third trigger keyword corresponding to the third request;
the direct action API to generate, based on the third trigger keyword and in response to the third request, a third action data structure with a parameter defining a third action;
the pooling component of the data processing system to determine, a second pooling parameter that indicates a level of overlap between the third action data structure and at least one of first action data structure, the second action data structure, and the pooling parameter; and
the pooling component of the data processing system to combine, based on the pooling parameter, the third action data structure into the pooled data structure.
5. The system of claim 1 , wherein the pooling parameter is a first pooling parameter, comprising:
the pooling component of the data processing system to determine, a second pooling parameter based on a third action data structure; and
the pooling component of the data processing system to modify the pooled data structure based on the second pooling parameter.
6. The system of claim 1 , wherein the level of overlap indicates a similarity metric between the first action data structure and the second action data structure.
7. The system of claim 1 , comprising:
the pooling component of the data processing system to determine the level of overlap between the first action data structure and the second action data structure based on subject matter indicated by the first action data structure and subject matter indicated by the second action data structure.
8. The system of claim 1 , wherein the first action includes a first plurality of sub-actions and the second action includes a second plurality of sub-actions.
9. The system of claim 1 , wherein the data processing system including the pooling component corresponds to a first entity, and the service provider computing device corresponds to a second entity different than the first entity.
10. The system of claim 1 , wherein the parameter defining the first action and the parameter defining the second action each indicate location data.
11. The system of claim 1 , wherein the parameter defining the first action and the parameter defining the second action each indicate time data.
12. The system of claim 1 , wherein the parameter defining the first action and the parameter defining the second action each identify common subject matter.
13. A method to manage voice activated threads in a voice activated data packet based computer network environment, comprising:
receiving, by a natural language processor component executed by a data processing system, via an interface of the data processing system, data packets comprising a first input audio signal detected by a sensor of a first client computing device;
parsing, by the natural language processor component, the first input audio signal to identify a first request and a first trigger keyword corresponding to the first request;
generating, by a direct action application programming interface (“API”) of the data processing system, based on the first trigger keyword and in response to the first request, a first action data structure with a parameter defining a first action;
receiving, by the natural language processor component, via the interface of the data processing system, data packets comprising a second input audio signal detected by a sensor of a second client computing device, and parsing, by the natural language processor component, the second input audio signal to identify a second request and a second trigger keyword corresponding to the second request;
generating, by the direct action API, based on the second trigger keyword and in response to the second request, a second action data structure with a parameter defining a second action;
determining, by a pooling component of the data processing system, based on a heuristic technique applied to the parameter of the first action data structure and the parameter of the second action data structure, a pooling parameter that indicates a level of overlap between the first action data structure and the second action data structure, the pooling parameter used to generate a pooled data structure that causes a reduction in processor or bandwidth utilization as compared to separate transmissions of the first action data structure and the second action data structure;
generating, based on the pooling parameter, the first action data structure and the second action data structure, the pooled data structure; and
transmitting, via a computer network, the pooled data structure to a service provider computing device to cause the service provider computing device to perform an operation defined by the pooled data structure and corresponding to the first action and the second action.
14. The method of claim 13 , wherein the service provider computing device is a first service provider computing device corresponding to a first entity, comprising:
transmitting, by the pooling component of the data processing system, via the computer network, a second pooled data structure to a second service provider computing device corresponding to a second entity to cause the second service provider computing device to perform an operation defined by the second pooled data structure.
15. The method of claim 13 , wherein the pooling parameter is a first pooling parameter, the pooled data structure is a first pooled data structure, and the service provider computing device is a first service provider computing device, comprising:
determining, by the pooling component of the data processing system, a second pooling parameter that indicates a level of overlap between a third action data structure and a fourth action data structure;
combining, based on the second pooling parameter, the third action data structure with the fourth action data structure into a second pooled data structure; and
transmitting, via the computer network, the second pooled data structure to a second service provider computing device to cause the second service provider computing device to perform an operation defined by the second pooled data structure and corresponding to the third action data structure and the fourth action data structure.
16. The method of claim 13 , wherein the pooling parameter is a first pooling parameter, comprising:
receiving, by the natural language processor component, via the interface of the data processing system, data packets comprising a third input audio signal detected by the sensor of the first client computing device; and
identifying, based on the third input audio signal, a third request and a third trigger keyword corresponding to the third request;
generating, based on the third trigger keyword and in response to the third request, a third action data structure with a parameter defining a third action;
determining a second pooling parameter that indicates a level of overlap between the third action data structure and at least one of first action data structure, the second action data structure, and the pooling parameter; and
combining based on the pooling parameter, the third action data structure into the pooled data structure.
17. The method of claim 13 , wherein the pooling parameter is a first pooling parameter, comprising:
determining, by the pooling component of the data processing system, a second pooling parameter that indicates a level of overlap between a third action data structure and at least one of first action data structure, the second action data structure, and the pooling parameter; and
combining, based on the pooling parameter, the third action data structure into the pooled data structure.
18. The method of claim 13 , comprising:
determining the level of overlap between the first action data structure and the second action data structure based on subject matter indicated by the first action data structure and subject matter indicated by the second action data structure.
19. The method of claim 13 , wherein the data processing system including the pooling component corresponds to a first entity, and the service provider computing device corresponds to a second entity different than the first entity.
20. The method of claim 13 , wherein the parameter defining the first action and the parameter defining the second action each indicate at least one of location data, time data, and common subject matter.
Priority Applications (20)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/395,707 US10013986B1 (en) | 2016-12-30 | 2016-12-30 | Data structure pooling of voice activated data packets |
US15/604,319 US11017428B2 (en) | 2008-02-21 | 2017-05-24 | System and method of data transmission rate adjustment |
DE112017000202.8T DE112017000202T5 (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice-activated data packets |
KR1020197024237A KR102268989B1 (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice activated data packets |
DE212017000042.2U DE212017000042U1 (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice-activated data packets |
KR1020217018632A KR20210076206A (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice activated data packets |
CN202111612483.7A CN114399999A (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice activated data packets |
CN201780001777.1A CN108541315B (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice activated data packets |
JP2017556893A JP6830444B2 (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice-activated data packets |
EP17768308.3A EP3360131B1 (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice activated data packets |
KR1020177031390A KR102015071B1 (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice active data packets |
AU2017384993A AU2017384993B2 (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice activated data packets |
EP19169787.9A EP3540728A1 (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice activated data packets |
GB1803296.1A GB2572533B (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice activated data packets |
PCT/US2017/049774 WO2018125304A1 (en) | 2016-12-30 | 2017-08-31 | Data structure pooling of voice activated data packets |
US16/018,750 US10423621B2 (en) | 2016-12-30 | 2018-06-26 | Data structure pooling of voice activated data packets |
US16/546,623 US10719515B2 (en) | 2016-12-30 | 2019-08-21 | Data structure pooling of voice activated data packets |
JP2020054359A JP2020115363A (en) | 2016-12-30 | 2020-03-25 | Data structure pooling of voice activated data packet |
US16/915,231 US11625402B2 (en) | 2016-12-30 | 2020-06-29 | Data structure pooling of voice activated data packets |
JP2022016301A JP7471333B2 (en) | 2016-12-30 | 2022-02-04 | Data structure pooling of voice-activated data packets |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/395,707 US10013986B1 (en) | 2016-12-30 | 2016-12-30 | Data structure pooling of voice activated data packets |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US201514831152A Continuation-In-Part | 2008-02-21 | 2015-08-20 |
Related Child Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/035,202 Continuation-In-Part US8065185B2 (en) | 2008-02-21 | 2008-02-21 | System and method of providing targeted advertisements from subscribers of directory services |
US15/604,319 Continuation-In-Part US11017428B2 (en) | 2008-02-21 | 2017-05-24 | System and method of data transmission rate adjustment |
US16/018,750 Continuation US10423621B2 (en) | 2016-12-30 | 2018-06-26 | Data structure pooling of voice activated data packets |
Publications (2)
Publication Number | Publication Date |
---|---|
US10013986B1 US10013986B1 (en) | 2018-07-03 |
US20180190299A1 true US20180190299A1 (en) | 2018-07-05 |
Family
ID=59887393
Family Applications (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/395,707 Active US10013986B1 (en) | 2008-02-21 | 2016-12-30 | Data structure pooling of voice activated data packets |
US16/018,750 Active US10423621B2 (en) | 2016-12-30 | 2018-06-26 | Data structure pooling of voice activated data packets |
US16/546,623 Active US10719515B2 (en) | 2016-12-30 | 2019-08-21 | Data structure pooling of voice activated data packets |
US16/915,231 Active 2037-12-13 US11625402B2 (en) | 2016-12-30 | 2020-06-29 | Data structure pooling of voice activated data packets |
Family Applications After (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/018,750 Active US10423621B2 (en) | 2016-12-30 | 2018-06-26 | Data structure pooling of voice activated data packets |
US16/546,623 Active US10719515B2 (en) | 2016-12-30 | 2019-08-21 | Data structure pooling of voice activated data packets |
US16/915,231 Active 2037-12-13 US11625402B2 (en) | 2016-12-30 | 2020-06-29 | Data structure pooling of voice activated data packets |
Country Status (9)
Country | Link |
---|---|
US (4) | US10013986B1 (en) |
EP (2) | EP3360131B1 (en) |
JP (3) | JP6830444B2 (en) |
KR (3) | KR102015071B1 (en) |
CN (2) | CN114399999A (en) |
AU (1) | AU2017384993B2 (en) |
DE (2) | DE112017000202T5 (en) |
GB (1) | GB2572533B (en) |
WO (1) | WO2018125304A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220309175A1 (en) * | 2021-03-29 | 2022-09-29 | Aipex Technologies, Inc. | Content management techniques for voice assistant |
Families Citing this family (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11017428B2 (en) | 2008-02-21 | 2021-05-25 | Google Llc | System and method of data transmission rate adjustment |
US10013986B1 (en) * | 2016-12-30 | 2018-07-03 | Google Llc | Data structure pooling of voice activated data packets |
US10771536B2 (en) * | 2009-12-10 | 2020-09-08 | Royal Bank Of Canada | Coordinated processing of data by networked computing resources |
US9339691B2 (en) | 2012-01-05 | 2016-05-17 | Icon Health & Fitness, Inc. | System and method for controlling an exercise device |
US9254409B2 (en) | 2013-03-14 | 2016-02-09 | Icon Health & Fitness, Inc. | Strength training apparatus with flywheel and related methods |
EP3974036A1 (en) | 2013-12-26 | 2022-03-30 | iFIT Inc. | Magnetic resistance mechanism in a cable machine |
US10433612B2 (en) | 2014-03-10 | 2019-10-08 | Icon Health & Fitness, Inc. | Pressure sensor to quantify work |
WO2015191445A1 (en) | 2014-06-09 | 2015-12-17 | Icon Health & Fitness, Inc. | Cable system incorporated into a treadmill |
WO2015195965A1 (en) | 2014-06-20 | 2015-12-23 | Icon Health & Fitness, Inc. | Post workout massage device |
US10391361B2 (en) | 2015-02-27 | 2019-08-27 | Icon Health & Fitness, Inc. | Simulating real-world terrain on an exercise device |
US10625137B2 (en) | 2016-03-18 | 2020-04-21 | Icon Health & Fitness, Inc. | Coordinated displays in an exercise device |
US10493349B2 (en) | 2016-03-18 | 2019-12-03 | Icon Health & Fitness, Inc. | Display on exercise device |
US10272317B2 (en) | 2016-03-18 | 2019-04-30 | Icon Health & Fitness, Inc. | Lighted pace feature in a treadmill |
US10671705B2 (en) | 2016-09-28 | 2020-06-02 | Icon Health & Fitness, Inc. | Customizing recipe recommendations |
KR102389041B1 (en) * | 2017-08-11 | 2022-04-21 | 엘지전자 주식회사 | Mobile terminal and method using machine learning for controlling mobile terminal |
KR20200004716A (en) * | 2018-07-04 | 2020-01-14 | 에스케이플래닛 주식회사 | Service apparatus, and control method thereof |
EP3752920A1 (en) * | 2019-05-06 | 2020-12-23 | Google LLC | Invoking functions of agents via digital assistant applications using address templates |
CN110705249B (en) * | 2019-09-03 | 2023-04-11 | 东南大学 | NLP library combined use method based on overlapping degree calculation |
US11158308B1 (en) * | 2019-11-27 | 2021-10-26 | Amazon Technologies, Inc. | Configuring natural language system |
CN112614282B (en) * | 2020-12-31 | 2022-07-15 | 深圳怡化电脑股份有限公司 | Service processing control method, device, electronic equipment and medium |
Citations (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5749069A (en) * | 1994-03-18 | 1998-05-05 | Atr Human Information Processing Research Laboratories | Pattern and speech recognition using accumulated partial scores from a posteriori odds, with pruning based on calculation amount |
US20030018479A1 (en) * | 2001-07-19 | 2003-01-23 | Samsung Electronics Co., Ltd. | Electronic appliance capable of preventing malfunction in speech recognition and improving the speech recognition rate |
US20030154072A1 (en) * | 1998-03-31 | 2003-08-14 | Scansoft, Inc., A Delaware Corporation | Call analysis |
US20060247913A1 (en) * | 2005-04-29 | 2006-11-02 | International Business Machines Corporation | Method, apparatus, and computer program product for one-step correction of voice interaction |
US20070282612A1 (en) * | 2006-05-31 | 2007-12-06 | Funai Electric Co., Ltd. | Electronic equipment |
US20080071536A1 (en) * | 2006-09-15 | 2008-03-20 | Honda Motor Co., Ltd. | Voice recognition device, voice recognition method, and voice recognition program |
US20080103781A1 (en) * | 2006-10-28 | 2008-05-01 | General Motors Corporation | Automatically adapting user guidance in automated speech recognition |
US20090210491A1 (en) * | 2008-02-20 | 2009-08-20 | Microsoft Corporation | Techniques to automatically identify participants for a multimedia conference event |
US20130073400A1 (en) * | 2011-09-15 | 2013-03-21 | Stephan HEATH | Broad and alternative category clustering of the same, similar or different categories in social/geo/promo link promotional data sets for end user display of interactive ad links, promotions and sale of products, goods and services integrated with 3d spatial geomapping and social networking |
US8453058B1 (en) * | 2012-02-20 | 2013-05-28 | Google Inc. | Crowd-sourced audio shortcuts |
US20130325449A1 (en) * | 2012-05-31 | 2013-12-05 | Elwha Llc | Speech recognition adaptation systems based on adaptation data |
US20140229184A1 (en) * | 2013-02-14 | 2014-08-14 | Google Inc. | Waking other devices for additional data |
US20140337028A1 (en) * | 2013-05-12 | 2014-11-13 | Shyh-Jye Wang | Message-triggered voice command interface in portable electronic devices |
US20140350938A1 (en) * | 2008-04-11 | 2014-11-27 | At&T Intellectual Property I, L.P. | System and method for detecting synthetic speaker verification |
US20150081288A1 (en) * | 2013-09-17 | 2015-03-19 | Electronics And Telecommunications Research Institute | Speech recognition device and the operation method thereof |
US20150106085A1 (en) * | 2013-10-11 | 2015-04-16 | Apple Inc. | Speech recognition wake-up of a handheld portable electronic device |
US20150221305A1 (en) * | 2014-02-05 | 2015-08-06 | Google Inc. | Multiple speech locale-specific hotword classifiers for selection of a speech locale |
US9275637B1 (en) * | 2012-11-06 | 2016-03-01 | Amazon Technologies, Inc. | Wake word evaluation |
US9318107B1 (en) * | 2014-10-09 | 2016-04-19 | Google Inc. | Hotword detection on multiple devices |
US20160180846A1 (en) * | 2014-12-17 | 2016-06-23 | Hyundai Motor Company | Speech recognition apparatus, vehicle including the same, and method of controlling the same |
US9424841B2 (en) * | 2014-10-09 | 2016-08-23 | Google Inc. | Hotword detection on multiple devices |
US9542941B1 (en) * | 2015-10-01 | 2017-01-10 | Lenovo (Singapore) Pte. Ltd. | Situationally suspending wakeup word to enable voice command input |
US20170069317A1 (en) * | 2015-09-04 | 2017-03-09 | Samsung Electronics Co., Ltd. | Voice recognition apparatus, driving method thereof, and non-transitory computer-readable recording medium |
US9653075B1 (en) * | 2015-11-06 | 2017-05-16 | Google Inc. | Voice commands across devices |
US20170147585A1 (en) * | 2014-07-22 | 2017-05-25 | Nuance Communications, Inc. | Systems and methods for speech-based searching of content repositories |
Family Cites Families (131)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7678023B1 (en) | 1995-06-22 | 2010-03-16 | Shea Michael J | Method for providing mental activity for an exerciser |
US6298218B1 (en) | 1996-12-18 | 2001-10-02 | Clubcom, Inc. | Combined advertising and entertainment system network |
US6574607B1 (en) | 1997-08-23 | 2003-06-03 | International Business Machines Corporation | Performing computer-based on-line commerce using an intelligent agent to put together a package of related items |
US7628730B1 (en) | 1999-07-08 | 2009-12-08 | Icon Ip, Inc. | Methods and systems for controlling an exercise apparatus using a USB compatible portable remote device |
AU6748900A (en) | 1999-07-30 | 2001-02-19 | Accenture Llp | A system, method and article of manufacture for e-commerce based performance modeling |
US6275806B1 (en) | 1999-08-31 | 2001-08-14 | Andersen Consulting, Llp | System method and article of manufacture for detecting emotion in voice signals by utilizing statistics for voice signal parameters |
US20020046084A1 (en) | 1999-10-08 | 2002-04-18 | Scott A. Steele | Remotely configurable multimedia entertainment and information system with location based advertising |
US8527345B2 (en) | 2000-01-06 | 2013-09-03 | Anthony Richard Rothschild | System and method for adding an advertisement to a personal communication |
US6505161B1 (en) | 2000-05-01 | 2003-01-07 | Sprint Communications Company L.P. | Speech recognition that adjusts automatically to input devices |
JP2003533770A (en) | 2000-05-05 | 2003-11-11 | 株式会社メガチップス | System and method for information acquisition and storage for deferred viewing |
US6684249B1 (en) | 2000-05-26 | 2004-01-27 | Sonicbox, Inc. | Method and system for adding advertisements over streaming audio based upon a user profile over a world wide area network of computers |
US6647269B2 (en) | 2000-08-07 | 2003-11-11 | Telcontar | Method and system for analyzing advertisements delivered to a mobile unit |
US6857007B1 (en) | 2000-08-30 | 2005-02-15 | Bloomfield Enterprises, Llc | Personal digital assistant facilitated communication system |
US6751475B1 (en) | 2000-10-19 | 2004-06-15 | At&T Wireless Services, Inc. | Shared-revenue billing system for transmission of wireless data from a vehicle |
US20030233278A1 (en) | 2000-11-27 | 2003-12-18 | Marshall T. Thaddeus | Method and system for tracking and providing incentives for tasks and activities and other behavioral influences related to money, individuals, technology and other assets |
US20020107027A1 (en) | 2000-12-06 | 2002-08-08 | O'neil Joseph Thomas | Targeted advertising for commuters with mobile IP terminals |
US20020087401A1 (en) | 2000-12-29 | 2002-07-04 | Gateway, Inc. | System and method for targeted advertising |
US7062469B2 (en) | 2001-01-02 | 2006-06-13 | Nokia Corporation | System and method for public wireless network access subsidized by dynamic display advertising |
US9183571B2 (en) | 2007-09-14 | 2015-11-10 | Qualcomm Incorporated | System and method for providing advertisement data to a mobile computing device |
US20020097193A1 (en) | 2001-01-23 | 2002-07-25 | Freecar Media | System and method to increase the efficiency of outdoor advertising |
US20020147638A1 (en) | 2001-04-05 | 2002-10-10 | International Business Machines Corporation | Business method for e-commerce through customized activity-based advertising |
DE10125909A1 (en) | 2001-05-28 | 2002-12-12 | Infineon Technologies Ag | Data transmission system with a high data transmission rate |
US7024211B1 (en) | 2001-12-26 | 2006-04-04 | Bellsouth Intellectual Property Corp. | System and method for managing text advertisements to mobile subscribers |
JP3870156B2 (en) | 2002-02-07 | 2007-01-17 | キヤノン株式会社 | Fiber plate and manufacturing method thereof, radiation imaging apparatus, and radiation imaging system |
US7013149B2 (en) | 2002-04-11 | 2006-03-14 | Mitsubishi Electric Research Laboratories, Inc. | Environment aware services for mobile devices |
US20040019654A1 (en) | 2002-07-29 | 2004-01-29 | Joseph Powers | Method and apparatus for exercise regimen administration |
JP2004192264A (en) * | 2002-12-10 | 2004-07-08 | Ntt Docomo Inc | Riding-together providing system, riding-together providing method, riding-together providing program and computer-readable record medium |
US7136658B2 (en) | 2002-12-10 | 2006-11-14 | International Business Machines Corporation | High-rate proximity detection with the ability to provide notification |
US20040192351A1 (en) | 2003-03-31 | 2004-09-30 | Duncan Daniel N. | Method and system for mobile display of context-based advertising content |
US20060236258A1 (en) | 2003-08-11 | 2006-10-19 | Core Mobility, Inc. | Scheduling of rendering of location-based content |
US20050038698A1 (en) | 2003-08-12 | 2005-02-17 | Lukose Rajan M. | Targeted advertisement with local consumer profile |
US8121898B2 (en) | 2003-10-06 | 2012-02-21 | Utbk, Inc. | Methods and apparatuses for geographic area selections in pay-per-call advertisement |
US7853255B2 (en) | 2004-04-16 | 2010-12-14 | Broadcom Corporation | Digital personal assistance via a broadband access gateway |
US20050245271A1 (en) | 2004-04-28 | 2005-11-03 | Sarosh Vesuna | System and method using location-aware devices to provide content-rich mobile services in a wireless network |
ES2796626T3 (en) | 2005-01-12 | 2020-11-27 | Invidi Tech Corp | Targeted Impression Model for Broadcast Network Resource Delivery |
US20060176289A1 (en) | 2005-02-05 | 2006-08-10 | Summerbrook Media Incorporated | Advertisement comprehension in mobile media |
US8768766B2 (en) | 2005-03-07 | 2014-07-01 | Turn Inc. | Enhanced online advertising system |
US7983690B2 (en) | 2005-03-24 | 2011-07-19 | General Motors Llc | Method and system for geographic boundary time triggering of communication with a mobile vehicle |
US20060224447A1 (en) | 2005-03-31 | 2006-10-05 | Ross Koningstein | Automated offer management using audience segment information |
US20060242012A1 (en) | 2005-04-22 | 2006-10-26 | Sumit Agarwal | Determining or scoring properties to solicit to join ad network using advertiser or aggregated advertiser interest |
US7451041B2 (en) | 2005-05-06 | 2008-11-11 | Facet Technology Corporation | Network-based navigation system having virtual drive-thru advertisements integrated with actual imagery from along a physical route |
US20060286989A1 (en) | 2005-05-20 | 2006-12-21 | Illion Brian E B | Geographical and calendar based advertising system and method |
US8732234B2 (en) | 2005-06-07 | 2014-05-20 | Yahoo! Inc. | Providing relevant non-requested content to a mobile device |
WO2007010611A1 (en) | 2005-07-21 | 2007-01-25 | Cirius Technologies, Inc. | Advertisement information display method, advertisement information display system, advertisement information display program, and advertisement information transmitting program |
WO2007022104A2 (en) | 2005-08-13 | 2007-02-22 | Adstreams Roi, Inc. | Enabling an advertiser to measure user viewing to and response to an advertisement |
US8195133B2 (en) | 2005-09-14 | 2012-06-05 | Jumptap, Inc. | Mobile dynamic advertisement creation and placement |
WO2007035959A2 (en) | 2005-09-23 | 2007-03-29 | Grape Technology Group Inc. | Enhanced directory assistance system and method including location and search functions |
US7505784B2 (en) | 2005-09-26 | 2009-03-17 | Barbera Melvin A | Safety features for portable electronic device |
US7904505B2 (en) | 2005-11-02 | 2011-03-08 | At&T Intellectual Property I, L.P. | Service to push author-spoken audio content with targeted audio advertising to users |
US7683252B2 (en) | 2005-11-23 | 2010-03-23 | Microsoft Corporation | Algorithm for providing music to influence a user's exercise performance |
US7996228B2 (en) | 2005-12-22 | 2011-08-09 | Microsoft Corporation | Voice initiated network operations |
US20070179359A1 (en) | 2006-01-10 | 2007-08-02 | Goodwin Amanda M | Healthy city living guide and related functionality for managing health |
CA2641853C (en) | 2006-02-10 | 2016-02-02 | Spinvox Limited | A mass-scale, user-independent, device-independent, voice messaging system |
US10803468B2 (en) | 2006-04-18 | 2020-10-13 | At&T Intellectual Property I, L.P. | Method and apparatus for selecting advertising |
JP2009536413A (en) | 2006-05-02 | 2009-10-08 | インビディ テクノロジーズ コーポレイション | Fuzzy logic based viewer identification for targeted asset delivery system |
US8571580B2 (en) | 2006-06-01 | 2013-10-29 | Loopt Llc. | Displaying the location of individuals on an interactive map display on a mobile communication device |
US8793066B2 (en) | 2006-06-27 | 2014-07-29 | Microsoft Corporation | Route monetization |
US7610151B2 (en) | 2006-06-27 | 2009-10-27 | Microsoft Corporation | Collaborative route planning for generating personalized and context-sensitive routing recommendations |
US20080004951A1 (en) | 2006-06-29 | 2008-01-03 | Microsoft Corporation | Web-based targeted advertising in a brick-and-mortar retail establishment using online customer information |
US20080005313A1 (en) | 2006-06-29 | 2008-01-03 | Microsoft Corporation | Using offline activity to enhance online searching |
US8725567B2 (en) | 2006-06-29 | 2014-05-13 | Microsoft Corporation | Targeted advertising in brick-and-mortar establishments |
US7617042B2 (en) | 2006-06-30 | 2009-11-10 | Microsoft Corporation | Computing and harnessing inferences about the timing, duration, and nature of motion and cessation of motion with applications to mobile computing and communications |
US20080004953A1 (en) | 2006-06-30 | 2008-01-03 | Microsoft Corporation | Public Display Network For Online Advertising |
US20080027799A1 (en) | 2006-07-28 | 2008-01-31 | Verizon Directory Services - West Inc. | Location-based advertising |
US9318108B2 (en) | 2010-01-18 | 2016-04-19 | Apple Inc. | Intelligent automated assistant |
US20080091518A1 (en) | 2006-09-28 | 2008-04-17 | Henry Eisenson | Adaptive cellular network advertising system |
US8165598B2 (en) | 2006-10-02 | 2012-04-24 | Mobitv, Inc. | Methods and apparatus for providing media on mobile devices |
US20080086360A1 (en) | 2006-10-06 | 2008-04-10 | Rajesh Kanapur | System and method for advertising via mobile devices |
US20080097836A1 (en) | 2006-10-23 | 2008-04-24 | Samuli Silanto | Advertisement presentment in an electronic device |
US10567909B2 (en) | 2006-10-26 | 2020-02-18 | Samsung Electronics Co., Ltd. | Wireless dissemination of environment aware information |
WO2008054715A2 (en) | 2006-10-31 | 2008-05-08 | Solicore, Inc. | Powered print advertisements, product packaging, and trading cards |
US9600959B2 (en) | 2007-01-09 | 2017-03-21 | Cfph, Llp | System for managing promotions |
US9754444B2 (en) | 2006-12-06 | 2017-09-05 | Cfph, Llc | Method and apparatus for advertising on a mobile gaming device |
US20080139181A1 (en) | 2006-12-08 | 2008-06-12 | Magellan Navigation, Inc. | Methods and apparatus for measuring the effectiveness of advertisements presented on a mobile navigation device |
EP2177010B1 (en) | 2006-12-13 | 2015-10-28 | Quickplay Media Inc. | Mobile media platform |
US20080153513A1 (en) | 2006-12-20 | 2008-06-26 | Microsoft Corporation | Mobile ad selection and filtering |
US8078196B2 (en) | 2007-01-31 | 2011-12-13 | At&T Intellectual Property I, Lp | Methods, systems and computer program products for providing information using an advertising message with a dynamic field |
US20080189215A1 (en) | 2007-02-01 | 2008-08-07 | Prototype Productions | Event driven advertising method and system |
US8352980B2 (en) | 2007-02-15 | 2013-01-08 | At&T Intellectual Property I, Lp | System and method for single sign on targeted advertising |
US8073460B1 (en) | 2007-03-08 | 2011-12-06 | Amazon Technologies, Inc. | System and method for providing advertisement based on mobile device travel patterns |
US8010134B2 (en) | 2007-03-14 | 2011-08-30 | Sprint Communications Company L.P. | Architecture for mobile advertising with location |
US20080242271A1 (en) | 2007-03-26 | 2008-10-02 | Kurt Schmidt | Electronic device with location-based and presence-based user preferences and method of controlling same |
US20080242231A1 (en) | 2007-03-29 | 2008-10-02 | Sony Ericsson Mobile Communications Ab | Updating Presence Based on Detecting User Activity |
US9092808B2 (en) | 2007-04-03 | 2015-07-28 | International Business Machines Corporation | Preferred customer marketing delivery based on dynamic data for a customer |
US8229458B2 (en) | 2007-04-08 | 2012-07-24 | Enhanced Geographic Llc | Systems and methods to determine the name of a location visited by a user of a wireless device |
US7861260B2 (en) | 2007-04-17 | 2010-12-28 | Almondnet, Inc. | Targeted television advertisements based on online behavior |
US7914419B2 (en) | 2007-05-29 | 2011-03-29 | Microsoft Corporation | Physical activity manager |
US10210531B2 (en) | 2007-06-18 | 2019-02-19 | Yellowpages.Com Llc | Systems and methods to facilitate the specification of a complex geographic area |
US20080319652A1 (en) | 2007-06-20 | 2008-12-25 | Radiofy Llc | Navigation system and methods for map navigation |
KR101322486B1 (en) * | 2007-06-28 | 2013-10-25 | 주식회사 케이티 | General dialogue service apparatus and method |
US20090005973A1 (en) | 2007-06-28 | 2009-01-01 | Salo Juha Heikki | Sponsored landmarks in navigation, couponing, parallel route calculation |
US8423003B2 (en) | 2007-07-11 | 2013-04-16 | Yahoo! Inc. | System for serving targeted advertisements over mobile messaging services |
US20090043657A1 (en) | 2007-08-06 | 2009-02-12 | Palm, Inc. | System and methods for selecting advertisements based on caller identifier information |
US20090048914A1 (en) | 2007-08-13 | 2009-02-19 | Research In Motion Limited | System and method for facilitating targeted mobile advertisement using pre-loaded ad content |
US8050690B2 (en) | 2007-08-14 | 2011-11-01 | Mpanion, Inc. | Location based presence and privacy management |
US20090055254A1 (en) | 2007-08-23 | 2009-02-26 | Yahoo! Inc. | Dynamic and interactive advertisements |
WO2009035698A1 (en) | 2007-09-12 | 2009-03-19 | Airkast, Inc. | Wireless device tagging system and method |
US20100299615A1 (en) | 2007-09-28 | 2010-11-25 | The Trustees Of Dartmouth College | System And Method For Injecting Sensed Presence Into Social Networking Applications |
US20090089166A1 (en) | 2007-10-01 | 2009-04-02 | Happonen Aki P | Providing dynamic content to users |
US20090132311A1 (en) | 2007-11-20 | 2009-05-21 | Theresa Klinger | Method and System for Monetizing User-Generated Content |
US7941557B2 (en) | 2007-11-28 | 2011-05-10 | Yahoo! Inc. | Dynamical routing for text messaging |
US8307029B2 (en) | 2007-12-10 | 2012-11-06 | Yahoo! Inc. | System and method for conditional delivery of messages |
US20090164299A1 (en) | 2007-12-21 | 2009-06-25 | Yahoo! Inc. | System for providing a user interface for displaying and creating advertiser defined groups of mobile advertisement campaign information targeted to mobile carriers |
US20090164300A1 (en) | 2007-12-21 | 2009-06-25 | Yahoo! Inc. | Application program interface and graphical user interface for providing a user interface for targeting mobile advertisements in a mobile marketing environment |
US20100312646A1 (en) | 2007-12-21 | 2010-12-09 | Yahoo! Inc. | System for serving advertisements targeted to geographic areas over mobile devices |
US20090197616A1 (en) | 2008-02-01 | 2009-08-06 | Lewis Robert C | Critical mass billboard |
US20090198538A1 (en) | 2008-02-05 | 2009-08-06 | Yahoo! Inc. | Mobile advertisement filtering |
US8731582B2 (en) | 2008-02-08 | 2014-05-20 | Trimble Navigation Limited | Method and system for generating targeted content |
US10013986B1 (en) * | 2016-12-30 | 2018-07-03 | Google Llc | Data structure pooling of voice activated data packets |
US8065185B2 (en) | 2008-02-21 | 2011-11-22 | At&T Intellectual Property I, L.P. | System and method of providing targeted advertisements from subscribers of directory services |
US10896426B2 (en) | 2008-05-09 | 2021-01-19 | International Business Machines Corporation | System and method for delivering distributed sensor based content to consumers |
US9305548B2 (en) | 2008-05-27 | 2016-04-05 | Voicebox Technologies Corporation | System and method for an integrated, multi-modal, multi-device natural language voice services environment |
US8589161B2 (en) * | 2008-05-27 | 2013-11-19 | Voicebox Technologies, Inc. | System and method for an integrated, multi-modal, multi-device natural language voice services environment |
US8145561B1 (en) | 2009-01-05 | 2012-03-27 | Sprint Communications Company L.P. | Phone usage pattern as credit card fraud detection trigger |
US9858925B2 (en) | 2009-06-05 | 2018-01-02 | Apple Inc. | Using context information to facilitate processing of commands in a virtual assistant |
US10276170B2 (en) | 2010-01-18 | 2019-04-30 | Apple Inc. | Intelligent automated assistant |
US9280610B2 (en) | 2012-05-14 | 2016-03-08 | Apple Inc. | Crowd sourcing information to fulfill user requests |
US10387448B2 (en) * | 2012-05-15 | 2019-08-20 | Splunk Inc. | Replication of summary data in a clustered computing environment |
US10417037B2 (en) * | 2012-05-15 | 2019-09-17 | Apple Inc. | Systems and methods for integrating third party services with a digital assistant |
DE212014000045U1 (en) | 2013-02-07 | 2015-09-24 | Apple Inc. | Voice trigger for a digital assistant |
US9538114B2 (en) | 2013-02-22 | 2017-01-03 | The Directv Group, Inc. | Method and system for improving responsiveness of a voice recognition system |
CN103426431B (en) * | 2013-07-24 | 2016-08-10 | 阳光凯讯(北京)科技有限公司 | The converged communication system of satellite network and terrestrial network system and dynamic acoustic code conversion method |
US20160255139A1 (en) * | 2016-03-12 | 2016-09-01 | Yogesh Chunilal Rathod | Structured updated status, requests, user data & programming based presenting & accessing of connections or connectable users or entities and/or link(s) |
US10445777B2 (en) | 2013-10-29 | 2019-10-15 | Verizon Patent And Licensing Inc. | Methods and systems for delivering electronic content to users in population based geographic zones |
CN103685504A (en) * | 2013-12-11 | 2014-03-26 | 南京大学 | Car sharing system based on Android platform and working method of car sharing system based on Android platform |
US10095878B2 (en) * | 2015-06-02 | 2018-10-09 | ALTR Solutions, Inc. | Internal controls engine and reporting of events generated by a network or associated applications |
US20170092278A1 (en) | 2015-09-30 | 2017-03-30 | Apple Inc. | Speaker recognition |
US9928840B2 (en) | 2015-10-16 | 2018-03-27 | Google Llc | Hotword recognition |
US9747926B2 (en) | 2015-10-16 | 2017-08-29 | Google Inc. | Hotword recognition |
US10691473B2 (en) | 2015-11-06 | 2020-06-23 | Apple Inc. | Intelligent automated assistant in a messaging environment |
US10192552B2 (en) | 2016-06-10 | 2019-01-29 | Apple Inc. | Digital assistant providing whispered speech |
-
2016
- 2016-12-30 US US15/395,707 patent/US10013986B1/en active Active
-
2017
- 2017-08-31 GB GB1803296.1A patent/GB2572533B/en active Active
- 2017-08-31 DE DE112017000202.8T patent/DE112017000202T5/en active Pending
- 2017-08-31 DE DE212017000042.2U patent/DE212017000042U1/en active Active
- 2017-08-31 KR KR1020177031390A patent/KR102015071B1/en active IP Right Grant
- 2017-08-31 CN CN202111612483.7A patent/CN114399999A/en active Pending
- 2017-08-31 KR KR1020217018632A patent/KR20210076206A/en not_active IP Right Cessation
- 2017-08-31 AU AU2017384993A patent/AU2017384993B2/en active Active
- 2017-08-31 JP JP2017556893A patent/JP6830444B2/en active Active
- 2017-08-31 KR KR1020197024237A patent/KR102268989B1/en active Application Filing
- 2017-08-31 WO PCT/US2017/049774 patent/WO2018125304A1/en active Application Filing
- 2017-08-31 EP EP17768308.3A patent/EP3360131B1/en active Active
- 2017-08-31 EP EP19169787.9A patent/EP3540728A1/en active Pending
- 2017-08-31 CN CN201780001777.1A patent/CN108541315B/en active Active
-
2018
- 2018-06-26 US US16/018,750 patent/US10423621B2/en active Active
-
2019
- 2019-08-21 US US16/546,623 patent/US10719515B2/en active Active
-
2020
- 2020-03-25 JP JP2020054359A patent/JP2020115363A/en active Pending
- 2020-06-29 US US16/915,231 patent/US11625402B2/en active Active
-
2022
- 2022-02-04 JP JP2022016301A patent/JP7471333B2/en active Active
Patent Citations (26)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5749069A (en) * | 1994-03-18 | 1998-05-05 | Atr Human Information Processing Research Laboratories | Pattern and speech recognition using accumulated partial scores from a posteriori odds, with pruning based on calculation amount |
US20030154072A1 (en) * | 1998-03-31 | 2003-08-14 | Scansoft, Inc., A Delaware Corporation | Call analysis |
US20030018479A1 (en) * | 2001-07-19 | 2003-01-23 | Samsung Electronics Co., Ltd. | Electronic appliance capable of preventing malfunction in speech recognition and improving the speech recognition rate |
US20060247913A1 (en) * | 2005-04-29 | 2006-11-02 | International Business Machines Corporation | Method, apparatus, and computer program product for one-step correction of voice interaction |
US20070282612A1 (en) * | 2006-05-31 | 2007-12-06 | Funai Electric Co., Ltd. | Electronic equipment |
US20080071536A1 (en) * | 2006-09-15 | 2008-03-20 | Honda Motor Co., Ltd. | Voice recognition device, voice recognition method, and voice recognition program |
US20080103781A1 (en) * | 2006-10-28 | 2008-05-01 | General Motors Corporation | Automatically adapting user guidance in automated speech recognition |
US20090210491A1 (en) * | 2008-02-20 | 2009-08-20 | Microsoft Corporation | Techniques to automatically identify participants for a multimedia conference event |
US20140350938A1 (en) * | 2008-04-11 | 2014-11-27 | At&T Intellectual Property I, L.P. | System and method for detecting synthetic speaker verification |
US20130073400A1 (en) * | 2011-09-15 | 2013-03-21 | Stephan HEATH | Broad and alternative category clustering of the same, similar or different categories in social/geo/promo link promotional data sets for end user display of interactive ad links, promotions and sale of products, goods and services integrated with 3d spatial geomapping and social networking |
US8453058B1 (en) * | 2012-02-20 | 2013-05-28 | Google Inc. | Crowd-sourced audio shortcuts |
US20130325449A1 (en) * | 2012-05-31 | 2013-12-05 | Elwha Llc | Speech recognition adaptation systems based on adaptation data |
US9275637B1 (en) * | 2012-11-06 | 2016-03-01 | Amazon Technologies, Inc. | Wake word evaluation |
US20140229184A1 (en) * | 2013-02-14 | 2014-08-14 | Google Inc. | Waking other devices for additional data |
US20140337028A1 (en) * | 2013-05-12 | 2014-11-13 | Shyh-Jye Wang | Message-triggered voice command interface in portable electronic devices |
US20150081288A1 (en) * | 2013-09-17 | 2015-03-19 | Electronics And Telecommunications Research Institute | Speech recognition device and the operation method thereof |
US20150106085A1 (en) * | 2013-10-11 | 2015-04-16 | Apple Inc. | Speech recognition wake-up of a handheld portable electronic device |
US20150221305A1 (en) * | 2014-02-05 | 2015-08-06 | Google Inc. | Multiple speech locale-specific hotword classifiers for selection of a speech locale |
US20170147585A1 (en) * | 2014-07-22 | 2017-05-25 | Nuance Communications, Inc. | Systems and methods for speech-based searching of content repositories |
US9318107B1 (en) * | 2014-10-09 | 2016-04-19 | Google Inc. | Hotword detection on multiple devices |
US20160217790A1 (en) * | 2014-10-09 | 2016-07-28 | Google Inc. | Hotword detection on multiple devices |
US9424841B2 (en) * | 2014-10-09 | 2016-08-23 | Google Inc. | Hotword detection on multiple devices |
US20160180846A1 (en) * | 2014-12-17 | 2016-06-23 | Hyundai Motor Company | Speech recognition apparatus, vehicle including the same, and method of controlling the same |
US20170069317A1 (en) * | 2015-09-04 | 2017-03-09 | Samsung Electronics Co., Ltd. | Voice recognition apparatus, driving method thereof, and non-transitory computer-readable recording medium |
US9542941B1 (en) * | 2015-10-01 | 2017-01-10 | Lenovo (Singapore) Pte. Ltd. | Situationally suspending wakeup word to enable voice command input |
US9653075B1 (en) * | 2015-11-06 | 2017-05-16 | Google Inc. | Voice commands across devices |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220309175A1 (en) * | 2021-03-29 | 2022-09-29 | Aipex Technologies, Inc. | Content management techniques for voice assistant |
Also Published As
Publication number | Publication date |
---|---|
US11625402B2 (en) | 2023-04-11 |
US10719515B2 (en) | 2020-07-21 |
EP3360131A1 (en) | 2018-08-15 |
GB201803296D0 (en) | 2018-04-11 |
US20200327121A1 (en) | 2020-10-15 |
GB2572533A8 (en) | 2019-10-30 |
CN114399999A (en) | 2022-04-26 |
US10423621B2 (en) | 2019-09-24 |
KR102015071B1 (en) | 2019-08-27 |
WO2018125304A1 (en) | 2018-07-05 |
GB2572533B (en) | 2022-06-22 |
JP2020115363A (en) | 2020-07-30 |
EP3540728A1 (en) | 2019-09-18 |
KR20190099353A (en) | 2019-08-26 |
US10013986B1 (en) | 2018-07-03 |
KR20180090729A (en) | 2018-08-13 |
US20180308493A1 (en) | 2018-10-25 |
JP7471333B2 (en) | 2024-04-19 |
DE112017000202T5 (en) | 2018-08-16 |
KR20210076206A (en) | 2021-06-23 |
CN108541315A (en) | 2018-09-14 |
JP2022070905A (en) | 2022-05-13 |
US20190377732A1 (en) | 2019-12-12 |
KR102268989B1 (en) | 2021-06-24 |
AU2017384993A1 (en) | 2019-05-23 |
DE212017000042U1 (en) | 2018-07-26 |
AU2017384993B2 (en) | 2020-02-06 |
JP6830444B2 (en) | 2021-02-17 |
JP2019505023A (en) | 2019-02-21 |
EP3360131B1 (en) | 2019-07-24 |
GB2572533A (en) | 2019-10-09 |
CN108541315B (en) | 2022-01-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11625402B2 (en) | Data structure pooling of voice activated data packets | |
US20210097997A1 (en) | Sequence dependent operation processing of packet based data message transmissions | |
US10893088B2 (en) | Sequence dependent data message consolidation in a voice activated computer network environment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: GOOGLE LLC, CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:GOOGLE INC.;REEL/FRAME:044567/0001 Effective date: 20170929 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |