US20180205569A1

US20180205569A1 - Method and apparatus for bidirectional control connecting hardware device action with url-based web navigation

Info

Publication number: US20180205569A1
Application number: US15/870,563
Authority: US
Inventors: Roger Wagner
Original assignee: Individual
Current assignee: Individual
Priority date: 2017-01-12
Filing date: 2018-01-12
Publication date: 2018-07-19
Also published as: US20180203711A1; US20210318892A1; US11023252B2; US20240028347A1; US11586449B2

Abstract

A method and apparatus, using a responsive electronic hardware device and Uniform Resource Locators (URLs) with an Internet browser display simultaneously, both to control web navigation through hardware device action and to control hardware device action through web navigation. The method exploits the basic functionality of a hardware micro controller and the universal and bidirectional effectiveness of URLs through a data table connecting URLs with actions and actions with URLs and including timing conditions. Embodiments employ the invention's understandability and ease of construction in educational, diagnostic, and entertainment applications.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No. 62/445,410, entitled “METHOD AND APPARATUS FOR BIDIRECTIONAL CONTROL CONNECTING HARDWARE DEVICE ACTION WITH URL-BASED WEB NAVIGATION,” filed on Jan. 12, 2017, the disclosure of which is incorporated herein in its entirety for all purposes.

BACKGROUND OF THE INVENTION

Background Art

Uniform Resource Locators (URLs), which are here taken as synonymous to Uniform Resource Identifiers (URIs), have resulted in everyday browser-related experiences, of which a few examples are web search, shopping, and filling out of forms. Their applicability to special-purpose electronic hardware (hereafter called “hardware” or “device”) has been more restricted, but is nevertheless common. Among obvious cases are webcams, security notification, and environmental monitoring which reports sensor data via the Internet.
Examples may be Internet-of-Things (IoT) systems, such as:

- (a) device to Internet to browser display, or an http-based web app, telling you that motion has been detected in your home, and,
- (b) browser display, or an http-based web app to Internet to device to turn on the lights in the home.

In (a), one type of prior art may have dedicated software that responds to the sensor condition with a new dedicated (non-browser) screen or display. In another type of prior art, the software displays data or messages within a screen on a webpage on the mobile device, such as weather station data being graphically displayed on a website.
In prior art of type (b) a person may use an app or an http-based web app to touch a screen display button to turn on lights, or it might even be automatically done by specific software, but there are not any existing methods where simply navigating to a new screen with a different URL on the device would turn on a light in the user's home.
In the device-to-webpage direction, the prior art only displays data on the webpage, it does not change the webpage based on the device-gathered data. In the webpage-to-device direction, prior art may be able to turn on something on the device through controls on the webpage, but the action does not occur as a direct result of simply being on the webpage.
It can be seen that the prior art discloses the display of data from a remote sensor on a given web-page, or the control of a remote sensor or actuator from the webpage, but that the URL of the webpage itself is not controlled by the sensors data, nor are actuators controlled by the act of changing the current URL of the display device. This deficiency in the prior art is corrected by the present invention.
A further example of the prior art involves a scenario of testing the security lights around a home. In all embodiments to date, the user clicks on different buttons presented on a webpage or in an app, and the lights come on, through methods very well-known to those skilled in the art. All these methods, however, involve the complication of programming the buttons and the response to the buttons.
In addition to the unidirectional URL-level control, bidirectional control is found in other prior art, e.g., use a webpage user interface to change the settings (direction, zoom, etc.) on a webcam, and with that in a fixed state, the webcam data flow is passed back as a display to the webpage. However, absent the method taught in this invention, the webpage does not change in response to the camera, and the camera does not change in response to changing the webpage being viewed.
The prior art approaches require programming of at least four pieces:

- the mobile or web app,
- the special-purpose communication on the app side that allows communication to the hardware,
- the hardware programming, and
- the special-purpose communication module that makes it possible for the hardware to communicate to the app.
  All of this must be done with dedication to a particular hardware and software combination, and it does not provide a general multi-application system that can be easily repurposed for different hardware sensor inputs and webpage displays.

The special-purpose communication can be designed to provide great speed if necessary, but it has the disadvantage of adding great complexity to the synergy of the hardware/web interface combination. That disadvantage is such as to restrict the development of a device with such a combination to very determined programmers. Thus the programming of a whole, responsive device involving both embedded hardware and an easily usable web interface becomes a specialty task.

SUMMARY

Accordingly, it is an object of the present invention to provide method and apparatus for bidirectional control connecting hardware device action with URL-based web navigation.
These and other objects and advantages of the present invention will become clear to those skilled in the art in view of the description of the best presently known mode of carrying out the invention and the industrial applicability of the preferred embodiment as described herein and as illustrated in the figures of the drawings.

BRIEF DESCRIPTION OF THE DRAWING(S)

The purposes and advantages of the present invention will be apparent from the following detailed description in conjunction with the appended tables and figures of drawings in which:

TBL. 1 is a description of an exemplary playlist;

FIG. 1 shows some timelines of a typical operation of a typical embodiment of the invention called the “HyperDuino”; and

FIG. 2 is a picture of the communicating pieces of one embodiment of the invention, HyperDuino For Chrome running an embedded device with a serial connection and connected via the Internet to servers from which URLs can draw videos or stills.

In the various figures of the drawings, like references are used to denote like or similar elements or steps.

DETAILED DESCRIPTION

A preferred embodiment of the present invention is a method and apparatus for bidirectional control connecting hardware device action with URL-based web navigation. As illustrated in the various drawings herein, preferred embodiments of the invention are depicted.
The invention is a system including a responsive electronic hardware device (the “hardware”), a browser with a user interface (the “browser”), a server or servers with content displayable by the browser (the “server”), and a software program (the “handler”) capable of:

- (a) communicating in either direction with the hardware,
- (b) of emitting URLs to the browser and server,
- (c) of receiving information from the browser about the current state of its web interface, including URLs displayed, anchors, selections, and optionally,
- (d) timing of changing media content such as videos.
  The browser and server are standard, and the hardware is specialized only in the sense that it is capable of monitoring sensors and inputs, and activating outputs, usually, but not limited to, actuators such as LEDs, motors, servos, buzzers, etc. and that the hardware communicates with the handler. The inventive content is focused on the combination of the browser, server, hardware and handler, as described below.

The handler is event-coded to comply with a table, herein called the “playlist,” which is a data table correlating hardware state input and browser URL location transmitted to the handler to actions performed by the handler. See TBL. 1. In what follows, the term “actuator” means the handler drives the hardware, either causing internal hardware state change or further effects within or beyond the hardware, or both, while the term “sensor” means the hardware that drives the handler, either as a result of the internal hardware state or conditions detected within or beyond the hardware, or both. The word “microcontroller” includes any electronics within the hardware.
Turning now to TBL. 1, this is a description of an exemplary playlist. The Fields 1a-zzz hold extra parameters, such as start and end times for a video which may have not been implicitly written in the URL, but which are still detectable by the Software while that URL is actively displayed in the browser. (these latter parameters may have been used in the prior art, but not within the context of the URL-associated I/O disclosed in this invention).
The Field 2: ResponseFlag (indicator) has three possible states. The first state is RespondNewURL. This initiates microcontroller output (actuator(s)) based on Fields 2a-zzz in response to the URL appearing in the browser display during the cycled review of the list, when the cycle before, that URL was not the active URL. That is, URL=(URL in list) AND (CurrentURL< >Past URL) AND (ActionTaken=False).
The second state is RespondLeaveURL. This initiates microcontroller output (actuator) based on Fields 2a-zzz in response to the URL NO LONGER appearing in the browser display during the cycled review of the list, when the cycle before, that URL WAS an active URL=found-in-list. That is, (CurrentURL< >Past URL) AND ActionTaken=False.
And the third state is RespondSense. This passes the URL to the browser in response to microcontroller output (sensor) meeting conditions.
The Fields Fields 2a-zzz are I/O channels, as appropriate to the microcontroller being used by the Software, and conditions to be tested for in case RespondSense or output to be actuated in cases RespondNewURL or RespondLeaveURL.
Causality goes from hardware state to URL in the Field 2 RespondSense case, and from web interface state to hardware state in the Field 2 RespondNewURL or RespondLeaveURL state.
The handler is simultaneously quickly responsive to web interface state, including URLs, anchors, selections, and timing such as time elapsed during a video, and to microcontroller (sensor) output to the handler. This can be accomplished by any sufficient programming construct, such as select( ) frequent enough polling, or event coding implied by the structure of a higher-level programming language like Javascript.
Since URLs can have multiple queries attached, including variable=value pairs, this kind of single-URL event coding can imply a rich branching structure and thus result in as arbitrarily complex event responses as are normally associated with deeply nested event-based coding with callbacks.
With local caching of data, internet latency, and even total disconnects, operation of the invention can still continue to operate satisfactorily by just passing the URL to the browser that itself uses cached data, and this is an example of how the URL-centric system provides an advantage that would be much more complex with custom-made computer code and applications.
Thus, the invention permits an arbitrarily complex event response to be coded as a flat table of URL responses. This table, which in one embodiment is called a′ “playlist,” responds simply to each specified URL, using its queries to guide the activation of actuators (defined earlier in the patent text), and also permits the display of any URL in the playlist in response to the output of sensors.

Example Embodiments

The first embodiment is the HyperDuino, an educational device. It builds on the simplicity inherent in the invention by introducing a synergistic simplicity to hardware construction, through the attachment of a shield called the “HyperDuino shield” to any Arduino or Arduino-equivalent device, such as the Funduino. The educational value of this is well described on the inventors web page [Wagner, Roger: “Hardware & Instructional Design—HyperDuino”; www.rogerwagner.com], for example:

- Physical classroom projects can now be made into interactive maker projects, but creating a two-way link between digital student-made content and physical student-created models.
- The HyperDuino solves key obstacles for those just entering the maker movement, by eliminating the need for breadboards and resistors, and combining with the HyperDuino for Chrome app to eliminate the need for scripted (written) coding, while still teaching the logic of programming.

The HyperDuino handler is instantiated as a Chrome OS app, written in Javascript. The creation of Chrome apps in Javascript is well known to web programmers skilled in the art. The HyperDuino handler is responsive to Web interface state, and it is also written so as to be responsive to asynchronous input from serial or other channels that are driven by the hardware.
After standard setup is complete and all communication channels both to the hardware and to the browser are functioning, the response of the HyperDuino handler is completely determined by its playlist, as described in TBL. 1. The HyperDuino handler continues operating until a HaltEvent is received, which can come either from internal programming, such as the operating session is done, or from an external signal, such as a user stop signal or arrival at an URL in the playlist, or the detection of a hardware state.
The transmission of URLs caused by the handler in the Field 2 RespondSense case can also include any web interface programming of which the browser is capable, and can fetch content from any accessible Internet server reached by the URL. The actuator response in the Field 2 RespondNewURL or RespondLeaveURL case can be driven by any knowledge of Web interface state that the browser is capable of sharing with the handler. Indirect immediate or delayed feedback responses, both as a result of hardware behavior and as a result of Web interface behavior, are possible and expected implications of the playlist.
The second embodiment presented here is an instructional and diagnostic tool for actual breadboard circuits, where analog and digital readings from the digital and analog inputs would directly lead to the display of informational and diagnostic webpages purely by virtual of how the existing playlist responded to the values.
For example, if the Arduino and a breadboard are being used to construct a circuit demonstrating a photocell, the HyperDuino software playlist will see “0” for a short circuit (for example wires crossed leading to the photocell), “1023” for an open circuit (for example, the photocell has not yet been inserted), and then an expected range when the photocell is inserted.
A lesson on a system like Google Slides may on slide N say “insert a photocell into this part of the circuit”. Until the photocell is inserted, the slide will not advance. When it is inserted, the slide will congratulate the user. Should a different item be inserted that is very different in expected resistance from a photocell, a different slide will be presented asking the operator to confirm and re-do the insertion of a photocell.
A third embodiment would be the application of the invention in program code on the server of a webpage itself, and not requiring a local application to be running. In that case, the webpage could display other webpages within a sub-window of the main webpage, as is commonly done with embedded videos and other html content. Code running on the server would then use the method taught in this invention to correlate the URL displayed in the sub-window with hardware inputs and outputs connected to the local computer by serial or other type of communication.

FIGURES AND EXPLANATION

FIG. 1 shows some timelines of a typical operation of a typical embodiment of the invention, hereafter called the “HyperDuino.”

- 101 is a timeline from left to right. 102 is the time spent in the HyperDuino Build stage, building the playlist. 103 is an idle time of indefinite length, during which the playlist is stored. 104 is the time spent in the HyperDuino Run stage, reading the playlist and acting upon it. 105 is a drawing representing the HyperDuino Builder, with its UI windows controlled by the builder, active during 102. 106 is a drawing representing the HyperDuino Handler, the standard browser and its screen and connection to the Internet, and its connected embedded device, all active during 104.
- 107 is a timeline within 104 during which a video running on the browser is synchronized by the HyperDuino Handler with action on the embedded device. 108 is the start of the video, 109 is an appropriate first time period, 110 is a command causing action such as lights or movement on the embedded device, 111 is an appropriate second time period, 112 is another command to the embedded device, 113 is an appropriate third time period, 114 is a third command to the embedded device.
- 115 is a timeline within 104 during which a video awaits selection by an embedded action. 116 is the start of the video, 117 is an appropriate first time period, 118 is a pause of the video, 119 is an indefinite time awaiting input from the embedded device (which could be a running state or a user button push, for instance), 120 is the awaited input, and 121 is the video resuming or another display taking its place if so programmed to be dependent on the input.
- 122 is a timeline within 104 during which the HyperDuino responds to a test result. 123 is a video, a still URL, or other display appropriate to the test result being unknown, 124 is a test result or state of the embedded device that may arrive unexpectedly and cause a branching action by the handler (time for branching not shown because it is very quick), and 125 is the chosen video or still URL appropriate to the test result detected.

FIG. 2 is a picture of the communicating pieces of one embodiment of the invention, HyperDuino For Chrome running an embedded device with a serial connection and connected via the Internet to two Servers from which URLs can draw videos or stills. This is during phase 104 of the timeline of FIG. 1.

- 201 is the embedded device. 202 is the HyperDuino Handler, and 206 is the standard Chrome browser, both running on the same system. 203, 204, and 205 are parts of the Handler: 203 is the playlist, which at this stage is read-only; 204 is HyperDuino For Chrome; and 205 is the HyperDuino Web Driver. 206 is the standard Chrome browser, 207 its screen. 208 is the Internet, and 209 a and 209 b are servers accessed by the browser through the Internet.

Bidirectional serial communication, in which either side can take the initiative, is 210 (embedded device to HyperDuino Web Driver) and 211 (HyperDuino Web Driver to embedded device). 212 represents HyperDuino For Chrome reading the playlist at will. 213 is data from HyperDuino For Chrome to HyperDuino Web Driver, and 214 is data from HyperDuino Web Driver to HyperDuino For Chrome, to effect the division of labor required by the OS and/or browser. 215 is an URL and/or timing sent from HyperDuino Web Driver to be acted upon by the browser, and 216 is the browser transmitting information about an URL to HyperDuino Web Driver. 217, 219 a, and 219 b represent the URL request going from browser through the Internet to a server, and 220 a, 220 b, and 218 represent the return of data by a server to the browser.
The embedded device and serial connection of 201, 210, and 211 are completely standard, as are the browser 206, its display 207, and the connection 217 and 218 and the Internet and servers beyond it. The breakdown of the HyperDuino Handler into HyperDuino For Chrome and HyperDuino Web Driver is due to requirements of the Chrome OS/browser. Other embodiments will use a different structure, but perform the same essential function.
While various embodiments have been described above, it should be understood that they have been presented by way of example only, and that the breadth and scope of the invention should not be limited by any of the above described exemplary embodiments, but should instead be defined only in accordance with the following claims and their equivalents.


TBE. 1. DESCRIPTION OF PLAYLIST

Field 1: URL (text)

	Fields 1a-zzz: Sub-field(s) to Field 1
	Field 2: ResponseFlag (indicator) as to whether this entry is,

Field 2 = RespondNewURL

or,

Field 2 = RespondLeaveURL

or,

Field 2 = RespondSense

	Fields 2a-zzz: Sub-fields to Field 2

Claims

1. A method for controlling a hardware device based on Uniform Resource Locator (URL)-based web navigation, the method comprising:

identifying a URL associated with a web browser;

determining, using an electronic lookup table, an action from a set of actions for a hardware device to perform based at least in part on the identified URL; and

causing the hardware device to perform the determined action.

2. The method of claim 1, the determined action comprises turning the hardware device on or off.

3. The method of claim 1, wherein the hardware device is a light source, and wherein the determined action comprises turning the light source on or off.

4. The method of claim 1, wherein the hardware device is a camera, and wherein the determined action comprises changing one or more settings of the camera.

5. The method of claim 4, wherein one or more settings of the camera comprises a combination of one or more of a direction or a zoom of the camera.

6. The method of claim 1, wherein the hardware device comprises one or more of an LED, a motor, or a servomechanism.

7. The method of claim 1, wherein the electronic lookup table correlates the URL and the determined action.

8. A method for navigating to a Uniform Resource Locator (URL) based on data corresponding to a hardware device, the method comprising:

receiving data indicative of a hardware state of a hardware device;

identifying, using an electronic lookup table, a URL location from a set of URLs based at least in part on the received data; and

causing a web browser to navigate to the identified URL.

9. The method of claim 8, wherein the hardware device comprises a sensor, and wherein the data corresponds to data sensed by the sensor.

10. The method of claim 9, wherein the sensor is a photocell.

11. The method of claim 8, wherein the hardware device comprises one or more of an LED, a motor, a servomechanism, or a buzzer.

12. The method of claim 8, wherein the hardware device is a camera, and wherein data of the hardware device corresponds to one or more settings of the camera.

13. The method of claim 12, the data of the hardware device corresponds to a change in direction or zoom of the camera.

14. The method of claim 8, wherein the electronic lookup table stored in a computer-readable medium.

15. The method of claim 8, wherein the electronic lookup table correlates the hardware state of the hardware device and the URL location.

16. A system for bidirectional control between hardware device action and URL-based web navigation, the system comprising:

a microcontroller in communication with at least one hardware device; and

one or more processors in communication with the microcontroller, and a non-transitory computer-readable storage medium storing computer-executable instructions that when executed by the one or more processors cause the one or more processors to:

receive data from the hardware device, wherein the data corresponds to the one or more actions,

monitor a web browser;

responsive to receiving first data, cause the web browser to navigate to a first URL, and

responsive to determining that the web page navigated to a second URL, causing the hardware device to perform a first action.

17. The system of claim 16, the first data corresponds to hardware state data of the hardware device.

18. The system of claim 16, wherein the hardware device is a light source, and wherein the first action turning the light source OFF.

19. The system of claim 16, wherein the hardware device is a light source, and wherein the first action turning the light source ON.