EP2766793B1 - Verbesserte zielauswahl für einen berührungsbasierte eingabeaktivierte benutzerschnittstelle - Google Patents
Verbesserte zielauswahl für einen berührungsbasierte eingabeaktivierte benutzerschnittstelle Download PDFInfo
- Publication number
- EP2766793B1 EP2766793B1 EP12840753.3A EP12840753A EP2766793B1 EP 2766793 B1 EP2766793 B1 EP 2766793B1 EP 12840753 A EP12840753 A EP 12840753A EP 2766793 B1 EP2766793 B1 EP 2766793B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- touch
- user interface
- graphical user
- toolbar
- display device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 claims description 43
- 239000013598 vector Substances 0.000 claims description 32
- 230000004044 response Effects 0.000 claims description 4
- 238000012545 processing Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000013461 design Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000013459 approach Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 5
- 239000003086 colorant Substances 0.000 description 5
- 230000000694 effects Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 241001422033 Thestylus Species 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000011960 computer-aided design Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000010079 rubber tapping Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000008685 targeting Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
Definitions
- the invention relates generally to a user interface and, more specifically, to enhanced target selection for a touch-based input enabled user interface.
- Touch screen displays are becoming increasingly common in consumer devices. For example, over the past decade, desktop monitors and hand-held devices have incorporated multi-touch surfaces with which users can interact to provide input information to the underlying devices.
- One interesting aspect of a multi-touch surface is the ability to control multiple degrees-of-freedom.
- conventional rectangle manipulations include translation, rotation, and scaling operations performed on a multi-touch surface by dragging, rotating, or expanding/contracting, respectively, one or more contact points.
- a user may interact with an application via a multi-touch surface by tapping, dragging, expanding, contracting, or rotating one or more contact points on the multi-touch surface. These operations are typically more efficient to perform on a multi-touch surface than with traditional input/output devices.
- touch screen displays have increased in popularity, many applications include a user interface that is not optimized for receiving touch-based inputs.
- an application may include a conventional menu bar and/or a conventional toolbar that includes small icons that are difficult for a user to accurately select. The user's ability to accurately touch a particular location compared to a conventional mouse input is decreased to the point where a user may select the wrong tool when attempted to interact with the interface using the touch screen display.
- Published US patent application US 2006/209016 A1 discloses a tablet input subsystem that receives pen and touch data, and also obtains keyboard and mouse data.
- the subsystem analyzes the data and determines which input device is currently active.
- the active device is mapped to an interaction model, whereby different user interface appearances, behaviors and the like may be presented to the user to facilitate improved interaction.
- a program may change the size of user interface elements to enable the user to more accurately scroll and make selections. Timing, tolerances and thresholds may change. Pen hovering can become active, and interaction events received at the same location can be handled differently.
- Published US patent application US 2009/058830 A1 discloses a portable device with a touch screen display that detects a finger-in-contact event at a first position on the touch screen display and identifies a user interface object associated with the first position.
- the portable device determines whether the identified user interface object is moveable across the touch screen display. For an identified object that is determined to be moveable across the touch screen display, the portable device detects a sequence of one or more first finger-dragging, a first finger-out-of-contact, one or more second finger-dragging, and a first finger-out-of-range events and performs a first operation associated with the identified moveable user interface object at least in part in accordance with the first and second finger-dragging events. For an identified not moveable object, the portable device detects a second finger-out-of-contact event and performs a second operation, different from the first operation in response to a second finger-out-of-contact event.
- One embodiment of the invention sets forth a computer-implemented method for displaying one or more touch-based graphical user interface elements according to claim 1.
- Another embodiment sets forth a computer-readable medium according to claim 5. Yet another embodiment sets forth a system for displaying one or more touch-based graphical user interface elements according to claim 6.
- One advantage of the disclosed technique is that a legacy application does not need to be redesigned to enable the touch-based graphical user interface elements to be implemented for the application.
- a programmer could build additional functionality into the legacy application or design a separate and distinct application extension that executes in parallel with the legacy application and provides an interface for the touch-based graphical user interface.
- a further advantage of the disclosed technique is that the touch-based GUI elements are displayed when the user intends to use them and hidden when the user provides input in a manner that is consistent with the original design of the application's graphical user interface. The techniques set forth below efficiently let a user switch between two different types of input.
- touch-based input may be more efficient, whereas other operations, such as entering text, may be more efficient using conventional input devices.
- the system allows both graphical user interfaces to be used substantially simultaneously with a single application.
- Figure 1 is a block diagram of a system 100 configured to implement one or more aspects of the present invention.
- System 100 may be a computer workstation, personal computer, video game console, or any other device suitable for practicing one or more embodiments of the present invention.
- system 100 includes one or more processing units, such as central processing unit (CPU) 102, and a system memory 104 communicating via a bus path that may include a memory bridge 105.
- CPU 102 includes one or more processing cores, and, in operation, CPU 102 is the master processor of system 100, controlling and coordinating operations of other system components.
- System memory 104 stores software applications and data for use by CPU 102.
- CPU 102 runs software applications and optionally an operating system.
- Memory bridge 105 which may be, e.g., a Northbridge chip, is connected via a bus or other communication path (e.g., a HyperTransport link) to an I/O (input/output) bridge 107.
- I/O bridge 107 which may be, e.g., a Southbridge chip, receives user input from one or more user input devices such as keyboard 108 or mouse 109 and forwards the input to CPU 102 via memory bridge 105.
- I/O bridge 107 may also be connected to other input devices such as a joystick, digitizer tablets, touch pads, touch screens, still or video cameras, motion sensors, and/or microphones (not shown).
- One or more display processors are coupled to memory bridge 105 via a bus or other communication path (e.g., a PCI Express®, Accelerated Graphics Port, or HyperTransport® link); in one embodiment, display processor 112 is a graphics subsystem that includes at least one graphics processing unit (GPU) and graphics memory.
- Graphics memory includes a display memory (e.g., a frame buffer) used for storing pixel data for each pixel of an output image. Graphics memory can be integrated in the same device as the GPU, connected as a separate device with the GPU, and/or implemented within system memory 104.
- Display processor 112 periodically delivers pixels to a display device 110.
- Display processor 112 can provide display device 110 with an analog or digital signal.
- Display device 110 may comprise a touch screen display device such as any conventional CRT or LED monitor with an integrated sensor that detects the presence and location of a user touching the display area of the monitor.
- Display device 110 may provide gesture recognition input to display processor 112 or CPU 102.
- a system disk 114 is also connected to I/O bridge 107 and may be configured to store content and applications and data for use by CPU 102 and display processor 112.
- System disk 114 provides non-volatile storage for applications and data and may include fixed or removable hard disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical, or solid state storage devices.
- a switch 116 provides connections between I/O bridge 107 and other components such as a network adapter 118 and various add-in cards 120 and 121.
- Network adapter 118 allows system 100 to communicate with other systems via an electronic communications network, and may include wired or wireless communication over local area networks and wide area networks such as the Internet.
- I/O bridge 107 may also be connected to I/O bridge 107.
- an audio processor may be used to generate analog or digital audio output from instructions and/or data provided by CPU 102, system memory 104, or system disk 114.
- Communication paths interconnecting the various components in Figure 1 may be implemented using any suitable protocols, such as Peripheral Component Interconnect (PCI), PCI Express (PCIe®), AGP (Accelerated Graphics Port), HyperTransport®, or any other bus or point-to-point communication protocol(s), and connections between different devices may use different protocols, as is known in the art.
- PCI Peripheral Component Interconnect
- PCIe® PCI Express
- AGP Accelerated Graphics Port
- HyperTransport® or any other bus or point-to-point communication protocol(s)
- display processor 112 incorporates circuitry optimized for graphics and video processing, including, for example, video output circuitry, and constitutes a graphics processing unit (GPU). In another embodiment, display processor 112 incorporates circuitry optimized for general purpose processing. In yet another embodiment, display processor 112 may be integrated with one or more other system elements, such as the memory bridge 105, CPU 102, and I/O bridge 107 to form a system on chip (SoC). In still further embodiments, display processor 112 is omitted and software executed by CPU 102 performs the functions of display processor 112.
- SoC system on chip
- Pixel data can be provided to display processor 112 directly from CPU 102.
- instructions and/or data representing a scene are provided to a render farm or a set of server computers, each similar to system 100, via network adapter 118 or system disk 114.
- the render farm generates one or more rendered images of the scene using the provided instructions and/or data. These rendered images may be stored on computer-readable media in a digital format and optionally returned to system 100 for display. Similarly, rendered images processed by display processor 112 may be output to other systems for display, stored in system disk 114, or stored on computer-readable media in a digital format.
- CPU 102 provides display processor 112 with data and/or instructions defining the desired output images, from which display processor 112 generates the pixel data of one or more output images.
- the data and/or instructions defining the desired output images can be stored in system memory 104 or a graphics memory within display processor 112.
- display processor 112 includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting shading, texturing, motion, and/or camera parameters for a scene.
- Display processor 112 can further include one or more programmable execution units capable of executing shader programs, tone mapping programs, and the like.
- Application 150 is stored in system memory 104.
- Application 150 may be any application configured to display a graphical user interface (GUI) on display device 110.
- GUI graphical user interface
- application 150 may be a common text editing application such as Microsoft® Word.
- the GUI included within application 150 is optimized for a conventional input devices such as keyboard 108 and mouse 109.
- an icon within a toolbar of application 150 may be 5 mm by 5 mm in size when displayed on display device 110, which is not easily selectable using touch-based input.
- Many conventional GUI elements are not easily selected or modified based on touch-based input.
- system memory 104 may be connected to CPU 102 directly rather than through a bridge, and other devices may communicate with system memory 104 via memory bridge 105 and CPU 102.
- display processor 112 may be connected to I/O bridge 107 or directly to CPU 102, rather than to memory bridge 105.
- I/O bridge 107 and memory bridge 105 may be integrated in a single chip.
- the particular components shown herein are optional. For instance, any number of add-in cards or peripheral devices might be supported.
- switch 116 is eliminated, and network adapter 118 and add-in cards 120, 121 connect directly to I/O bridge 107.
- Figures 2A and 2B are a front-view and side-view, respectively, of a touch screen desktop environment 200, according to one embodiment of the present invention.
- the touch screen desktop environment 200 includes, without limitation, a display device 110 and a keyboard 108.
- the display device 110 is positioned vertically on a desktop surface 210 facing the end-user.
- the keyboard 108 may be positioned on top of the desktop surface 210 between the display device 110 and the user.
- the display device 110 may be a non-conventional display such as a projector focused on a wall that includes a proximity sensor, an iPadTM or other tablet computer, or a mobile device such as a cell phone with a touch sensitive screen.
- the front surface of display device 110 implements a touch sensitive area that includes multi-touch gesture recognition capabilities.
- An end-user may use one or more fingers to perform various gestures that initiate operations such as rotate, scale, and zoom.
- the display device 110 includes the capability to display graphics and video on the multi-touch surface.
- display device 110 may integrate a sensor 225 that enables the display device 110 to detect the presence of an object above the surface of the screen. For example, some manufacturers have produced a capacitive screen that integrates an array of optical sensors that can detect a location of a finger approximately 20 mm above the surface of the multi-touch surface. In other embodiments, display device 110 may integrate any other technically feasible technique for detecting the location of a finger in proximity to the touch-sensitive surface.
- two or more cameras may be placed along the perimeter of display device 110 and image recognition software could be used to estimate the coordinates of an object in proximity to the multi-touch surface.
- image recognition software could be used to estimate the coordinates of an object in proximity to the multi-touch surface.
- a depth sensitive camera such as a CMOS sensor integrated with an infrared laser projector may be used to judge distance from the touch-sensitive surface.
- the sensor 225 may detect an approximate location of the finger 250 relative to the multi-touch surface.
- sensor 225 determines an x-coordinate, y-coordinate, and z-coordinate associated with the user's finger 250 relative to the display device 110.
- a motion vector 260 may be calculated that indicates the direction of motion towards a point on the multi-touch surface.
- sensor 225 is configured to track the position of a user's finger 250 at a slow rate such as 100 Hz (i.e., the sensor 225 calculates x, y, and z coordinates for the finger 100 times a second).
- CPU 102 may be configured to generate a motion vector 260 based on the last two sampled coordinates of sensor 225.
- the motion vector 260 may be generated based on more than two samples of sensor 225.
- CPU 102 may generate motion vector 260 based on three or more samples of sensor 225, selecting the motion vector 260 that produces the minimum mean square error for the sampled position coordinates.
- Motion vector 260 may be scaled such that motion vector 260 intersects with a plane associated with the surface of display device 110.
- sensor 225 may be configured to track any object that is proximate to the surface of the display device 110. For example, instead of a finger, a user may use a stylus to provide touch based input.
- display device 110 is enabled to detect multi-touch input.
- the user may touch two or more points on the surface of display device 110 to provide multi-touch input.
- sensor 225 may be configured to track multiple objects, such as multiple fingers.
- the motion vector 260 may be determined based on the object closest to the display device 110.
- motion vector 260 may be based on some combination of two or more motion vectors calculated for multiple objects. For example, the motion vector 260 could be calculated based on an average of two or more motion vectors.
- Figure 3 illustrates a graphical user interface 300 generated by an application 150 and displayed on a display device 110, according to one embodiment of the present invention.
- application 150 is a text editing application that implements GUI 300 and includes a menu bar 310 and a toolbar 320.
- application 150 may be some other type of application such as a computer-aided design (CAD) application, a spreadsheet application, or an email application.
- CAD computer-aided design
- Menu bar 310 includes such menu items as a File menu, an Edit menu, and a Help menu.
- a drop down menu may appear that provides additional GUI elements associated with various functions of application 150.
- GUI elements such as Open, Save, and Save As that, when selected by a user, are configured to cause dialog boxes to be displayed for performing various file system operations.
- Toolbar 320 includes a plurality of GUI elements as well.
- toolbar 320 includes one or more toolbar icons 322 as well as one or more combo-boxes 324 (i.e., a combination of a single line text box and a drop-down list box).
- Such toolbar icons 322 may include a save icon, a print icon, and various style icons related to a text font such as a bold icon, an italics icon, and an underline icon.
- the combo-boxes 324 may include GUI elements for selecting a particular font or font size. It will be appreciated that other types of GUI elements may be included in menu bar 310 or toolbar 320.
- the particular design of GUI 300 shown in Figure 3 is provided to illustrate various techniques of the present invention and should not be construed to limit the invention to this particular GUI.
- the scope of this disclosure contemplates other GUI designs as well as other types of GUI elements known in the art.
- other types of GUI elements include radio buttons and drop-down lists.
- GUI 300 is designed for receiving input from conventional input devices such as keyboard 108 and mouse 109.
- GUI elements may be designed for ease of use with touch-based input such as by designing toolbar icons 322 with a minimum size that corresponds to an accuracy provided by touch-based sensing technology. For example, by studying the characteristics of a user base and comparing the characteristics to the sensitivity of a capacitive touch screen sensor, a GUI designer may determine that a user can select a point on the screen to an accuracy of 10 mm. Therefore, any GUI element that is designed for touch-based input should be designed with an accuracy expectation of 10 mm. Such GUI elements may be too large to effectively provide a user with all of the commonly used functions on one screen.
- application 150 could be configured to display GUI elements optimized for touch-based input when the user provides touch-based input and, conversely, to display the conventional GUI elements when the user provides conventional input using a keyboard and mouse. Two such techniques for displaying touch-based GUI elements to a user are discussed below.
- Figure 4 illustrates how application 150 detects a finger 250 proximate to the display device 110, according to one embodiment of the present invention.
- a user may launch application 150 on computer system 100.
- Application 150 causes GUI 300 to be displayed on display device 110.
- application 150 is configured to receive input via keyboard 108 and mouse 109, application 150 may also be configured to receive touch-based input from display device 110.
- a user may provide touch-based input by touching the touch-sensitive area of display device 110 with the user's finger 250.
- display device 110 includes a sensor 225 that may sense when the user's finger 250 is proximate to the touch-sensitive area of display device 110.
- display device 110 may include the capability to detect a finger 250 when the finger is within 20 mm of the surface of display device 110.
- sensor 225 detects the presence of a finger 250
- sensor 225 transmits a position of the finger 250 to application 150.
- Sensor 225 may be configured to transmit a new position to application 150 at regular intervals until sensor 225 can no longer detect finger 250, indicating that the finger 250 is no longer approaching the touch-sensitive area of display device 110.
- Application 150 may be configured to generate motion vector 260 based on the coordinates received from sensor 225.
- motion vector 260 may be the vector directed from a first set of coordinates received from sensor 225 to a second set of coordinates received from sensor 225. Consequently, motion vector 260 represents the direction of finger 250 as a user moves to touch display device 110.
- application 150 may calculate a target location 450 associated with GUI 300 that represents a predicted point on display device 110 that the user is attempting to touch.
- application 150 may be configured to determine the intersection of motion vector 260 with the surface of display device 110 (i.e., the plane formed by all points with a z-coordinate of 0).
- Target location 450 therefore, represents an estimation of which element of GUI 300 the user is targeting.
- the target location 450 may be more accurate the closer the user's finger 250 gets to the surface of display device 110.
- the accuracy of the estimation algorithm is affected by the sampling frequency of the position information, the number of samples that effect the calculation, and the relative distance of the finger from the surface of the display device 110.
- Figure 5 illustrates a technique for displaying GUI elements optimized for touch-based input, according to one embodiment of the present invention.
- application 150 when application 150 detects finger 250 proximate to display device 110, application 150 causes one or more touch-based GUI elements to be displayed over GUI 300.
- application 150 causes touch-based icons 512 and 514 to be overlaid on top of GUI 300.
- Touch-based icon 512 corresponds to a bold icon in GUI 300 and touch-based icon 514 corresponds to an underline icon in GUI 300.
- a triggering event causes application 150 to display the touch-based GUI elements.
- the triggering event may be detected whenever a user first touches the surface of display device 110.
- the touch-based GUI elements could be displayed if the coordinate of the target location 450 is proximate to an element of GUI 300 associated with one of the touch-based GUI elements.
- the triggering event could be detected whenever sensor 225 detects a finger 250 proximate to the display device 110.
- the triggering event may compare the z-coordinate of the position of the finger 250 to a threshold value. For example, the triggering event may occur whenever the finger 250 comes within 10 mm of the display device 110.
- triggering event may occur only when the finger 250 is positioned with a threshold value from the display device 110 within a particular region of the screen. For example, the triggering event occurs when the finger 250 is within 10 mm of the display device 110 and the y-coordinate of the position is also above a second threshold (e.g., the finger is approaching the top half of the screen).
- multiple triggering events may be defined based on two or more threshold values. Each triggering event may cause the touch-based GUI elements to be displayed at a different scale factor. For example, when the finger 250 is less than a first threshold distance from the surface of the display device 110 a first triggering event causes touch-based GUI elements to be displayed at a 50% scale, which may still be larger than the conventional GUI elements of GUI 300. When the finger 250 is then less than a second threshold distance, smaller than the first threshold distance, from the surface of the display device 110 a second triggering event causes touch-based GUI elements to be displayed at 100% scale.
- the triggering event does not require target prediction (i.e., determining an estimated intersection between the motion vector 260 of the finger 250 and the display device 110).
- target prediction i.e., determining an estimated intersection between the motion vector 260 of the finger 250 and the display device 110.
- the set of touch-based GUI elements is displayed at a position overlaid on GUI 300 that is independent of any type of target location 450.
- the touch-based icons are displayed on top of associated GUI elements in GUI 300.
- the touch-based icons may be opaque or partially transparent.
- the touch-based GUI elements may be displayed in different positions on the display device 110. For example, all touch based icons could be aligned to the upper-left of the GUI 300.
- a user could determine where the touch-based GUI elements are positioned on the display device 110 using input from mouse 109 or touch-based input.
- a tooltip associated with the touch-based GUI elements may be displayed if the user hovers his finger 250 above the surface of the display device after the triggering event. For example, a tooltip that includes the text "Bold" may appear next to touch-based icon 512.
- Touch-based icons 512 and 514 are included in the set of touch-based GUI elements displayed by application 150 when a triggering event is detected.
- the set of touch-based GUI elements may be configured automatically by application 150 or may be manually configured by a user of application 150.
- the set of touch-based GUI elements may correspond to the most frequently used GUI elements of GUI 300.
- the set of touch-based GUI elements may be empty when application 150 is first launched and a user manually selects which touch-based GUI elements to add to the set, each touch-based GUI element corresponding to one of the GUI elements in GUI 300.
- Figure 6 illustrates a procedure for associating conventional GUI elements with a set of touch-based GUI elements displayed by the application 150, according to one embodiment of the present invention.
- Touch-based combo-box 612 is now included in the set of touch-based GUI elements displayed by the application 150.
- a user touches display device 110 at a location proximate to combo-box 324 of GUI 300 (i.e., the font size combo-box). The user indicates that a touch-based GUI element corresponding to combo-box 324 should be added to the set of touch-based GUI elements through some combination of keystrokes via keyboard 108 or mouse input via mouse 109.
- the user could hit a hotkey such as "F8" to indicate that a touch-based GUI element should be added to the set of touch-based GUI elements.
- a hotkey such as "F8"
- each of the touch-based GUI elements in the set of touch-based GUI elements is displayed over GUI 300.
- a user may specify which conventional GUI elements of GUI 300 to include in the set of touch-based GUI elements displayed by application 150.
- a user may tap display device 110 close to a desired GUI element that the user wishes to include in the set of touch-based GUI elements. Then, the user may click and drag the cursor via the mouse 109 to define the desired size of the touch-based GUI element.
- a user may tap the touch-based GUI element followed by selecting the touch-based GUI element with mouse 109.
- the technique for displaying touch-based GUI elements in response to detecting a triggering event enables efficient use of touch-based input with legacy applications.
- the touch-based GUI elements only need to be displayed when the system detects that the user is intending to use touch-based input.
- the increased size of the touch-based GUI elements may make optimizing all available GUI elements impractical because the available screen space is limited.
- Another technique for displaying a touch-based GUI is described below.
- Figure 7A illustrates how application 150 determines a target location 450 based on the motion vector 260 associated with a finger 250 proximate to display device 110, according to one embodiment of the present invention.
- application 150 determines a target location 450 based on the intersection of motion vector 260 with the touch-sensitive surface of display device 110.
- application 150 determines two consecutive positions of finger 250 (i.e., each position includes an x-coordinate, y-coordinate, and z-coordinate for finger 250) and calculates a motion vector 260 based on the difference between the two positions.
- the motion vector 260 is then scaled such that the motion vector intersects with the touch-sensitive surface of display device 110 (i.e., corresponding to the plane where the z-coordinates equal 0).
- the point of intersection between motion vector 260 and the multi-touch surface of display device 110 is determined to be the target location 450.
- application 150 determines a portion of a toolbar to expand into a touch-based toolbar. For example, as shown in Figure 7A , application 150 selects a portion 710 of toolbar 320 that corresponds to a center of expansion 715 based on target location 450. The center of expansion is the top of toolbar 320 at the x-coordinate corresponding to target location 450. Therefore, when the touch-based toolbar is displayed, the touch-based toolbar appears to expand down from the top of toolbar 320.
- Figure 7B illustrates a touch-based toolbar 750 overlaid on top of toolbar 320 in GUI 300, according to one embodiment of the present invention.
- toolbar 750 is twice as large as toolbar 320.
- Toolbar 750 is sized to accommodate touch-based input. It will be appreciated that toolbar 750 only includes touch-based GUI elements corresponding to the GUI elements within the portion 710 of toolbar 320 that corresponds to the center of expansion 715 in Figure 7A .
- TouchZoom enables access to each GUI element in GUI 300 proximate to an estimated target location 450. This operation ensures expansion of the desired touch-based GUI element, even with a slight error in the estimation of the target location 450.
- the toolbar 320 may be expanded relative to the position of target location 450 on display device 110.
- Figures 8A and 8B illustrate an approach to expanding toolbar 320, according to one embodiment of the present invention.
- target location 450 is positioned at the extreme left edge of display device 110.
- the portion 810 of toolbar 320 is located entirely to the right of center of expansion 815.
- the left edge of toolbar 850 corresponds to the left edge of the portion 810 of toolbar 320.
- toolbar 320 is expanded down and to the right to display touch-based GUI elements corresponding to the various GUI elements in toolbar 320.
- toolbar 320 is expanded using different techniques based on the position of target location 450 relative to different zones of display device 110. If target position 450 is located within an edge zone defined as an area close to the left edge of display device 110, then the center of expansion 815 of toolbar 320 is located on the left edge of display device 110 and toolbar 850 expands both down and to the right. Similarly, if target position 450 is located within another edge zone defined as an area close to the right edge of display device 110, then the center of expansion 815 of toolbar 320 is located on the right edge of display device 110 and toolbar 850 expands both down and to the left.
- the edge zone may be defined to be anywhere within 50 mm of the edge of the display device 110. If target location 450 is located within 50 mm of an edge of display device 110, then the center of expansion 815 of toolbar 320 is located at the edge of display device 110 regardless of where within the edge zone the target location 450 is actually located.
- a buffer zone may be defined between the edge zone and the center zone, where, similar to the center zone, the center of expansion 815 corresponds to the target location 450 but, unlike in the center zone, toolbar 320 is expanded non-uniformly around the center of expansion 815.
- portion 810 would be selected such that 25% of portion 810 is located to the left of the center of expansion 815 and 75% of portion 810 is located to the right of the center of expansion 815.
- the percentage of portion 810 on one side of the center of expansion 815 compared to the other side of the center of expansion ranges linearly over the buffer zone from 0% if the target location 450 is within the edge zone to 50% if the target location 450 is within the center zone.
- Figure 9 illustrates a different algorithm for estimating a center of expansion for toolbar 320, according to yet another embodiment of the present invention.
- Toolbar 320 is divided into a plurality of groups of GUI elements.
- toolbar 320 includes four distinct groups of GUI elements, group 910, group 920, group 930, and group 940.
- application 150 may determine a center of expansion 815 based on which group target location 450 is closest to.
- application 150 determines whether target location 450 intersects with one of groups of GUI elements (910, 920, etc.).
- Target location 450 intersects with one of the groups of GUI elements if target location 450 is located at a pixel position in display device 110 within a group of GUI elements of toolbar 320.
- application 150 determines the closest distance between target location 450 and each of the groups of GUI elements (i.e., any pixel within the group of GUI elements of toolbar 320).
- the set of pixels within the group of GUI elements includes pixels that are included in the GUI elements within the group as well as pixels that are included in the background of toolbar 320 between the GUI elements within the group.
- toolbar 320 is expanded based on a center of expansion associated with that group of GUI elements. For example, a center of expansion for group 910 may be located at the top, left edge of group 910. A center of expansion for groups 920 and 930 may be located at the top, center of the groups of GUI elements. A center of expansion for group 940 may be located at the top, right edge of group 940. In one embodiment, the entirety of toolbar 320 is expanded similar to toolbar 750 or toolbar 850.
- FIG. 10 illustrates an approach for expanding only a portion of toolbar 320, according to another example embodiment of the present invention.
- touch-based toolbar 1050 corresponds to a target location 450 that intersects with or is closest to group of GUI elements 920 in toolbar 320 (as shown in Figure 9 ).
- the center of expansion of the group of GUI elements 920 is the top, centered pixel of the group of GUI elements 920.
- touch-based toolbar 1050 is located at a position that is approximately centered on the center of expansion of group 920.
- toolbar 320 may include GUI elements that are configured as functionality reduction controls.
- a functionality reduction control may be configured to have certain functions when the user provides input using conventional input devices.
- the GUI element may transition into a touch-based GUI element with the same footprint as the original GUI element of toolbar 320 but with reduced functionality.
- a toolbar includes a color palette that is configured as a functionality reduction control, then the color palette may be replaced by a plurality of color selection icons that allow the user to pick only a subset of preconfigured colors from the color palette. The user can quickly choose one of the preconfigured colors using touch-based input (such as 8 primary colors) or choose one of the available colors from the color palette using convention mouse input (such as 256 available colors).
- an application extension may be designed that interfaces with application 150, in effect displaying a separate GUI on top of the GUI displayed by application 150.
- the separate GUI captures touch-based input received from display device 110, processes the touch-based input to determine a particular GUI element in application 150 associated with the touch-based input, and transmit corresponding conventional input associated with the particular GUI element to application 150.
- the application extension intercepts touch-based commands and converts the touch-based commands into conventional commands suitable for application 150.
- FIG 11 is a flowchart of method steps 1100 for displaying touch-based graphical user interface elements in an application 150, according to one example embodiment of the present invention.
- the method steps are described in conjunction with the systems of Figures 1-10 , persons skilled in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the inventions.
- the method 1100 begins at step 1110, where a user launches an application 150 that includes instructions to display a GUI 300 that is designed for conventional input devices such as keyboard 108 or mouse 109.
- the application 150 is executed by CPU 102, which is coupled to a display device 110 that includes a touch-screen capability.
- CPU 102 determines whether a user intends to provide input via the display device 110.
- sensor 225 detects that a user's finger 250 is proximate to a surface of the display device 110 configured to receive touch-based input.
- the sensor 225 senses a distance of the finger 250 from the surface. If the finger 250 is within a threshold distance, then CPU 102 determines that the user intends to provide touch-based input.
- sensor 225 tracks the position of the finger 250 and determines a target location 450 associated with the GUI 300. If the target location 450 corresponds to a particular location of the GUI 300, then CPU 102 determines that the user intends to provide touch-based input. A technique for determining the target location is described below in connection with Figure 12 . If CPU 102 determines that a user does not intend to provide touch-based input, then the method waits at step 1112. However, if CPU 102 determines that a user intends to provide touch based input, then method 1100 proceeds to step 1114.
- CPU 102 displays one or more touch-based GUI elements on top of GUI 300 on display device 110.
- a set of touch-based GUI elements are pre-defined, each touch-based GUI element corresponding to one of the conventional GUI elements of GUI 300.
- CPU 102 determines that the user intends to provide touch based input, CPU 102 causes the set of touch-based elements to be displayed.
- a user may manually configure which conventional GUI elements should be associated with a touch-based GUI element in the set of touch-based GUI elements that are displayed.
- CPU 102 causes a touch-based toolbar 750 to be displayed over GUI 300.
- the touch-based toolbar 750 corresponds to an expanded version of toolbar 320 of GUI 300.
- the touch-based toolbar 750 is expanded based on a center of expansion 715 determined from the target location 450 calculated by CPU 102.
- CPU 102 determines whether a user intends to provide input via conventional input devices such as keyboard 108 and mouse 109. In one embodiment, CPU 102 determines that a user intends to provide input via conventional input devices when sensor 225 no longer senses that a user's finger 250 is proximate to the display device 110. In another embodiment, CPU 102 may wait until a threshold time has elapsed since sensor 225 last sensed the user's finger 250. In yet another embodiment, CPU 102 determines that a user intends to provide input via conventional input devices when a user begins providing such input by either typing with keyboard 108 or by moving mouse 109. If CPU 102 determines that a user intends to provide input via conventional input devices, then method 1100 proceeds to step 1118 where CPU 102 hides the one or more touch-based GUI elements such that the conventional GUI 300 is displayed on display device 110.
- conventional input devices such as keyboard 108 and mouse 109.
- Figure 12 is a flowchart of method steps 1200 for estimating a target location 450 associated with a graphical user interface 300, according to one example embodiment of the present invention.
- the method steps are described in conjunction with the systems of Figures 1-10 , persons skilled in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the inventions.
- the method 1200 begins at step 1210, where CPU 102 detects that an object is located proximate to display device 110.
- sensor 225 is configured to sense whenever an object, such as a finger 250, is located proximate to the surface of display device 110.
- Display device 110 is configured to transmit a signal to CPU 102 that includes information about the location of any object proximate to display device 110.
- CPU 102 calculates a motion vector 260 associated with the object.
- CPU 102 samples positions of an object over time via information received from display device 110 (i.e., sensor 225 of display device 110).
- Motion vector 260 may be a vector between two or more positions of the object at different points of time.
- CPU 102 determines a target location 450 that indicates an estimated location in GUI 300 that the user intends to touch.
- target location 450 is based on the intersection of motion vector 260 with the surface of display device 110.
- CPU 102 determines a center of expansion (715, 815) based on the target location 450.
- the center of expansion is the point at the top of toolbar 320 that corresponds to the x-coordinate of the target location 450.
- the center of expansion is a point that is defined based on the location of target location 450 within GUI 300. For example, the center of expansion, if the target location 450 is within an edge zone, is located at the top of toolbar 320 at the left edge or right edge of GUI 300.
- the techniques disclosed above provide an efficient technique for providing GUI elements optimized for touch-based input in connection with an application that is designed for conventional input received from a keyboard and a mouse.
- the touch-based GUI elements are displayed when the computer system detects that a user is about to switch from conventional input devices to touch-sensitive input devices.
- the touch-based GUI elements are hidden when the user provides input with the conventional input devices such that the touch-based GUI elements do not distract from the application's normal operation.
- One advantage of the disclosed technique is that a legacy application does not need to be redesigned to enable the touch-based GUI elements to be implemented for the application.
- a programmer could build additional functionality into the legacy application or design a separate and distinct application extension that executes in parallel with the legacy application and provides an interface for the touch-based GUI.
- a further advantage of the disclosed technique is that the touch-based GUI elements are displayed when the user intends to use them and hidden when the user provides input in a manner that is consistent with the original design of the application's GUI. This provides a transparent system that efficiently lets a user switch between two different types of input. For some operations, such as selecting options or scrolling through a document, touch-based input may be more efficient, whereas other operations, such as entering text, may be more efficient using conventional input devices. The system allows both GUIs to be used substantially simultaneously.
- aspects of the present invention may be implemented in hardware or software or in a combination of hardware and software.
- One embodiment of the present invention may be implemented as a program product for use with a computer system.
- the program(s) of the program product define functions of the embodiments (including the methods described herein) and can be contained on a variety of computer-readable storage media.
- Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, flash memory, ROM chips or any type of solid-state non-volatile semiconductor memory) on which information is permanently stored; and (ii) writable storage media (e.g., floppy disks within a diskette drive or hard-disk drive or any type of solid-state random-access semiconductor memory) on which alterable information is stored.
- non-writable storage media e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, flash memory, ROM chips or any type of solid-state non-volatile semiconductor memory
- writable storage media e.g., floppy disks within a diskette drive or hard-disk drive or any type of solid-state random-access semiconductor memory
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
Claims (6)
- Computerimplementiertes Verfahren (1100) zum Anzeigen von einem oder mehreren berührungsbasierten grafischen Benutzeroberflächenelementen, wobei das Verfahren umfasst:Ausführen (1110) einer Anwendung, die bewirkt, dass eine grafische Benutzeroberfläche (300) auf einer Berührungsbildschirmanzeige (110), die mit einer Vorrichtung (100) assoziiert ist, angezeigt wird, wobei die grafische Benutzeroberfläche dazu konfiguriert ist, mit mindestens einer von einer Tastatur- und einer Maus-Eingabevorrichtung (108, 109) zusammenzuwirken;Bestimmen (1112), dass ein Benutzer eine Eingabe in die Vorrichtung mittels der Berührungsbildschirmanzeige bereitstellt, umfassend:Erfassen, dass ein Objekt (250) sich in der Nähe einer Oberfläche der Berührungsbildschirmanzeige befindet;Berechnen eines Bewegungsvektors (260), der mit dem Objekt (250) assoziiert ist und der eine Bewegungsbahn des Objekts in Bezug zu der Oberfläche der Berührungsbildschirmanzeige angibt; undBerechnen einer Zielstelle (450) eines Schnittpunkts des Bewegungsvektors mit der Oberfläche, wobei die Zielstelle einer Stelle innerhalb einer herkömmlichen Symbolleiste (320) der grafischen Benutzeroberfläche entspricht;als Reaktion auf ein Erfassen, dass ein Objekt sich in der Nähe und innerhalb eines Grenzabstands von der Oberfläche der Berührungsbildschirmanzeige befindet, Anzeigen (1114) von einer oder mehreren berührungsbasierten grafischen Benutzeroberflächenelementen einer berührungsbasierten Symbolleiste (750) an einem oberen Teil der grafischen Benutzeroberfläche, wobei ein Expansionszentrum (715, 815) auf der Basis der Zielstelle bestimmt wird und wobei die berührungsbasierte Symbolleiste, die das eine oder die mehreren berührungsbasierten grafischen Benutzeroberflächenelemente umfasst, auf der Basis des Expansionszentrums expandiert wird;Bereitstellen (1116) einer Eingabe mittels mindestens einer von einer Tastatur- und einer Maus-Eingabevorrichtung undals Reaktion auf ein Bereitstellen der Eingabe mittels mindestens einer von einer Tastatur- und einer Maus-Eingabevorrichtung Ausblenden (1118) des einen oder der mehreren berührungsbasierten grafischen Benutzeroberflächenelemente der berührungsbasierten Symbolleiste,wobei jedes berührungsbasierte grafische Benutzeroberflächenelement einem anderen herkömmlichen grafischen Benutzeroberflächenelement, das innerhalb der grafischen Benutzeroberfläche enthalten ist, entspricht.
- Verfahren (1100) nach Anspruch 1, wobei jedes berührungsbasierte grafische Benutzeroberflächenelement der berührungsbasierten Symbolleiste (750) angezeigt wird auf der Basis von:einem Berühren der Berührungsbildschirmanzeige (110) an einer Stelle, die mit einem bestimmten herkömmlichen GUI-Element assoziiert ist, durch den Benutzer undeinem Spezifizieren einer Größe für das berührungsbasierte grafische Benutzeroberflächenelement mittels der Maus-Eingabevorrichtung (109) durch den Benutzer,wobei das berührungsbasierte grafische Benutzeroberflächenelement dem bestimmten herkömmlichen grafischen Benutzeroberflächenelement entspricht.
- Verfahren (1100) nach Anspruch 1, wobei das Expansionszentrum eine Ankerstelle für die berührungsbasierte Symbolleiste in Bezug auf eine herkömmliche Symbolleiste, die innerhalb der grafischen Benutzeroberfläche (300) angezeigt wird, definiert.
- Verfahren (1100) nach Anspruch 1, wobei das Expansionszentrum (715, 815) mit einem vordefinierten Punkt für eine Gruppe von grafischen Benutzeroberflächenelementen innerhalb der herkömmlichen Symbolleiste assoziiert ist.
- System (100) zum Anzeigen von einem oder mehreren berührungsbasierten grafischen Benutzeroberflächenelementen, wobei das System umfasst:eine Berührungsbildschirmanzeige (110);einen Speicher (104), der eine Anwendung (150) beinhaltet, die bewirkt, dass eine grafische Benutzeroberfläche (300) auf der Berührungsbildschirmanzeige angezeigt wird, wobei die grafische Benutzeroberfläche dazu konfiguriert ist, mit mindestens einer von einer Tastatur- und einer Maus-Eingabevorrichtung (108, 109) zusammenzuwirken; undeinen Prozessor (112), der dazu konfiguriert ist, die wie in den Ansprüchen 1 bis 4 vorgetragenen Arbeitsschritte durchzuführen.
- Computerlesbares Speichermedium, das ein Programm enthält, das bei Ausführung durch einen Prozessor des Systems nach Anspruch 5 ein Verfahren zum Anzeigen von einem oder mehreren berührungsbasierten grafischen Benutzeroberflächenelementen durchführt, wobei das Verfahren die Arbeitsschritte nach einem der Ansprüche 1 bis 4 umfasst.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/274,253 US10684768B2 (en) | 2011-10-14 | 2011-10-14 | Enhanced target selection for a touch-based input enabled user interface |
PCT/US2012/059876 WO2013055997A1 (en) | 2011-10-14 | 2012-10-12 | Enhanced target selection for a touch-based input enabled user interface |
Publications (3)
Publication Number | Publication Date |
---|---|
EP2766793A1 EP2766793A1 (de) | 2014-08-20 |
EP2766793A4 EP2766793A4 (de) | 2015-07-29 |
EP2766793B1 true EP2766793B1 (de) | 2020-05-20 |
Family
ID=48082467
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP12840753.3A Active EP2766793B1 (de) | 2011-10-14 | 2012-10-12 | Verbesserte zielauswahl für einen berührungsbasierte eingabeaktivierte benutzerschnittstelle |
Country Status (4)
Country | Link |
---|---|
US (1) | US10684768B2 (de) |
EP (1) | EP2766793B1 (de) |
CN (1) | CN104011629B (de) |
WO (1) | WO2013055997A1 (de) |
Families Citing this family (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8863232B1 (en) | 2011-02-04 | 2014-10-14 | hopTo Inc. | System for and methods of controlling user access to applications and/or programs of a computer |
US20130104039A1 (en) * | 2011-10-21 | 2013-04-25 | Sony Ericsson Mobile Communications Ab | System and Method for Operating a User Interface on an Electronic Device |
US8872794B2 (en) | 2011-10-28 | 2014-10-28 | Sony Corporation | Information processing apparatus |
US9928562B2 (en) | 2012-01-20 | 2018-03-27 | Microsoft Technology Licensing, Llc | Touch mode and input type recognition |
US20130191779A1 (en) * | 2012-01-20 | 2013-07-25 | Microsoft Corporation | Display of user interface elements based on touch or hardware input |
US20130271362A1 (en) * | 2012-04-12 | 2013-10-17 | Sergiy Lozovsky | Method, System and Program Product for Enhancing a Graphical User Interface |
US9419848B1 (en) | 2012-05-25 | 2016-08-16 | hopTo Inc. | System for and method of providing a document sharing service in combination with remote access to document applications |
US8713658B1 (en) | 2012-05-25 | 2014-04-29 | Graphon Corporation | System for and method of providing single sign-on (SSO) capability in an application publishing environment |
US8856907B1 (en) | 2012-05-25 | 2014-10-07 | hopTo Inc. | System for and methods of providing single sign-on (SSO) capability in an application publishing and/or document sharing environment |
US9400575B1 (en) * | 2012-06-20 | 2016-07-26 | Amazon Technologies, Inc. | Finger detection for element selection |
US9213436B2 (en) | 2012-06-20 | 2015-12-15 | Amazon Technologies, Inc. | Fingertip location for gesture input |
US20140035827A1 (en) * | 2012-07-31 | 2014-02-06 | Elwha LLC, a liability company of the State of Delaware | Touch screen display compensated for a carrier-induced motion |
US9239812B1 (en) | 2012-08-08 | 2016-01-19 | hopTo Inc. | System for and method of providing a universal I/O command translation framework in an application publishing environment |
KR102091710B1 (ko) * | 2012-08-28 | 2020-04-14 | 삼성전자주식회사 | 좌표 측정 장치 및 그 제어 방법 |
US8886372B2 (en) * | 2012-09-07 | 2014-11-11 | The Boeing Company | Flight deck touch-sensitive hardware controls |
US9589538B2 (en) * | 2012-10-17 | 2017-03-07 | Perceptive Pixel, Inc. | Controlling virtual objects |
US20140108979A1 (en) * | 2012-10-17 | 2014-04-17 | Perceptive Pixel, Inc. | Controlling Virtual Objects |
JP5875510B2 (ja) * | 2012-12-10 | 2016-03-02 | 株式会社ソニー・コンピュータエンタテインメント | 電子機器、メニュー表示方法 |
US20140219634A1 (en) | 2013-02-05 | 2014-08-07 | Redux, Inc. | Video preview creation based on environment |
JP5991538B2 (ja) * | 2013-02-20 | 2016-09-14 | 富士ゼロックス株式会社 | データ処理装置、データ処理システム及びプログラム |
US10572124B2 (en) * | 2013-05-20 | 2020-02-25 | Citrix Systems, Inc. | Bound based contextual zoom |
US20140354554A1 (en) * | 2013-05-30 | 2014-12-04 | Microsoft Corporation | Touch Optimized UI |
US9182904B2 (en) * | 2013-11-11 | 2015-11-10 | Lenovo (Singapore) Pte. Ltd. | Cues based on location and context for touch interface |
KR102123092B1 (ko) * | 2013-11-21 | 2020-06-15 | 삼성전자주식회사 | 지문 인식 방법 및 그 전자 장치 |
US9282283B2 (en) * | 2014-01-29 | 2016-03-08 | Microsoft Technology Licensing, Llc | Detecting patterns traced on a screen of a user device |
US10048856B2 (en) | 2014-12-30 | 2018-08-14 | Microsoft Technology Licensing, Llc | Configuring a user interface based on an experience mode transition |
CN106293444B (zh) * | 2015-06-25 | 2020-07-03 | 小米科技有限责任公司 | 移动终端、显示控制方法及装置 |
EP3356918A1 (de) * | 2015-09-29 | 2018-08-08 | Telefonaktiebolaget LM Ericsson (publ) | Berührungsbildschirmvorrichtung und verfahren dafür |
US10048837B2 (en) | 2016-02-16 | 2018-08-14 | Google Llc | Target selection on a small form factor display |
CN106445383A (zh) * | 2016-09-26 | 2017-02-22 | 艾体威尔电子技术(北京)有限公司 | 一种用于触摸屏上输入密码的方法 |
CN106951080A (zh) * | 2017-03-16 | 2017-07-14 | 联想(北京)有限公司 | 用于控制虚拟物体的交互方法和装置 |
US10909288B2 (en) * | 2017-12-26 | 2021-02-02 | Autodesk, Inc. | Techniques for applying generative design to the configuration of mechanical assemblies |
KR102539578B1 (ko) * | 2018-02-19 | 2023-06-05 | 삼성전자주식회사 | 어플리케이션의 기능을 매핑하는 방법 및 그 전자 장치 |
JP2019211826A (ja) * | 2018-05-31 | 2019-12-12 | キヤノン株式会社 | 制御装置、制御システム、制御方法およびプログラム |
CN114764301B (zh) * | 2021-01-13 | 2024-05-24 | 深圳市汉森软件股份有限公司 | 打印机控制界面交互方法、装置、控制板及打印机 |
WO2023108475A1 (en) | 2021-12-15 | 2023-06-22 | Citrix Systems, Inc. | Application hotspot on endpoint device |
Family Cites Families (48)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2648558B2 (ja) * | 1993-06-29 | 1997-09-03 | インターナショナル・ビジネス・マシーンズ・コーポレイション | 情報選択装置及び情報選択方法 |
US7760187B2 (en) * | 2004-07-30 | 2010-07-20 | Apple Inc. | Visual expander |
US6232957B1 (en) * | 1998-09-14 | 2001-05-15 | Microsoft Corporation | Technique for implementing an on-demand tool glass for use in a desktop user interface |
US6286104B1 (en) * | 1999-08-04 | 2001-09-04 | Oracle Corporation | Authentication and authorization in a multi-tier relational database management system |
US7434177B1 (en) * | 1999-12-20 | 2008-10-07 | Apple Inc. | User interface for providing consolidation and access |
US6683627B1 (en) * | 2000-09-28 | 2004-01-27 | International Business Machines Corporation | Scroll box controls |
US7730401B2 (en) | 2001-05-16 | 2010-06-01 | Synaptics Incorporated | Touch screen with user interface enhancement |
DE602004011907D1 (de) | 2003-03-10 | 2008-04-03 | Koninkl Philips Electronics Nv | Anzeige mit mehrfachen Ansichten |
KR101042099B1 (ko) | 2003-08-21 | 2011-06-16 | 마이크로소프트 코포레이션 | 공중 지점을 이용한 포커스 관리방법, 시스템 및 컴퓨터 판독 가능 매체 |
JP5090161B2 (ja) * | 2004-06-29 | 2012-12-05 | コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ | グラフィカルユーザインタフェースの多階層表示 |
US20060189278A1 (en) * | 2005-02-24 | 2006-08-24 | Research In Motion Limited | System and method for making an electronic handheld device more accessible to a disabled person |
US7802202B2 (en) * | 2005-03-17 | 2010-09-21 | Microsoft Corporation | Computer interaction based upon a currently active input device |
US20070057911A1 (en) * | 2005-09-12 | 2007-03-15 | Sina Fateh | System and method for wireless network content conversion for intuitively controlled portable displays |
US7600194B2 (en) * | 2005-12-02 | 2009-10-06 | Microsoft Corporation | Start menu operation for computer user interface |
US20080229254A1 (en) * | 2006-03-24 | 2008-09-18 | Ervin-Dawson Warner | Method and system for enhanced cursor control |
US8564544B2 (en) * | 2006-09-06 | 2013-10-22 | Apple Inc. | Touch screen device, method, and graphical user interface for customizing display of content category icons |
CN101529874A (zh) | 2006-09-06 | 2009-09-09 | 苹果公司 | 用于具有触摸屏显示器的便携式多功能设备的电话呼入管理 |
US8375304B2 (en) * | 2006-11-01 | 2013-02-12 | Skyfire Labs, Inc. | Maintaining state of a web page |
US20080165145A1 (en) * | 2007-01-07 | 2008-07-10 | Scott Herz | Portable Multifunction Device, Method, and Graphical User Interface for Interpreting a Finger Swipe Gesture |
US8665225B2 (en) * | 2007-01-07 | 2014-03-04 | Apple Inc. | Portable multifunction device, method, and graphical user interface for interpreting a finger gesture |
US8018441B2 (en) * | 2007-06-11 | 2011-09-13 | Samsung Electronics Co., Ltd. | Character input apparatus and method for automatically switching input mode in terminal having touch screen |
JP2009026155A (ja) * | 2007-07-20 | 2009-02-05 | Toshiba Corp | 入力表示装置および移動無線端末装置 |
US8689139B2 (en) * | 2007-12-21 | 2014-04-01 | Adobe Systems Incorporated | Expandable user interface menu |
US20090172606A1 (en) * | 2007-12-31 | 2009-07-02 | Motorola, Inc. | Method and apparatus for two-handed computer user interface with gesture recognition |
US8745514B1 (en) * | 2008-04-11 | 2014-06-03 | Perceptive Pixel, Inc. | Pressure-sensitive layering of displayed objects |
US8826181B2 (en) * | 2008-06-28 | 2014-09-02 | Apple Inc. | Moving radial menus |
US9658765B2 (en) * | 2008-07-31 | 2017-05-23 | Northrop Grumman Systems Corporation | Image magnification system for computer interface |
US8074181B2 (en) * | 2008-09-15 | 2011-12-06 | Microsoft Corporation | Screen magnifier panning model with dynamically resizable panning regions |
US8237666B2 (en) * | 2008-10-10 | 2012-08-07 | At&T Intellectual Property I, L.P. | Augmented I/O for limited form factor user-interfaces |
US8413066B2 (en) * | 2008-11-06 | 2013-04-02 | Dmytro Lysytskyy | Virtual keyboard with visually enhanced keys |
CN102239068B (zh) * | 2008-12-04 | 2013-03-13 | 三菱电机株式会社 | 显示输入装置 |
DE112010001445T5 (de) * | 2009-03-31 | 2012-10-25 | Mitsubishi Electric Corporation | Anzeigeeingabegerät |
US20100293499A1 (en) * | 2009-05-14 | 2010-11-18 | Microsoft Corporation | Rendering to a device desktop of an adaptive input device |
US20100328232A1 (en) * | 2009-06-30 | 2010-12-30 | Wood James A | Touch Screen Cursor Presentation Preview Window |
CN101937267A (zh) | 2009-07-03 | 2011-01-05 | 北京宏景捷讯网络技术股份有限公司 | 模拟鼠标输入的方法及装置 |
US20110057886A1 (en) * | 2009-09-10 | 2011-03-10 | Oliver Ng | Dynamic sizing of identifier on a touch-sensitive display |
CN201522696U (zh) | 2009-09-14 | 2010-07-07 | 武汉传威光控科技有限公司 | 基于光学传感的触摸控制装置 |
US9218126B2 (en) * | 2009-09-21 | 2015-12-22 | Extreme Reality Ltd. | Methods circuits apparatus and systems for human machine interfacing with an electronic appliance |
US8766928B2 (en) * | 2009-09-25 | 2014-07-01 | Apple Inc. | Device, method, and graphical user interface for manipulating user interface objects |
US8261211B2 (en) * | 2009-10-01 | 2012-09-04 | Microsoft Corporation | Monitoring pointer trajectory and modifying display interface |
US8423897B2 (en) * | 2010-01-28 | 2013-04-16 | Randy Allan Rendahl | Onscreen keyboard assistance method and system |
JP4892068B2 (ja) * | 2010-01-29 | 2012-03-07 | 株式会社東芝 | 情報処理装置及び画像表示方法 |
JP5160579B2 (ja) * | 2010-03-08 | 2013-03-13 | 株式会社エヌ・ティ・ティ・ドコモ | 表示装置及び画面表示方法 |
JP5768347B2 (ja) * | 2010-09-07 | 2015-08-26 | ソニー株式会社 | 情報処理装置、情報処理方法およびコンピュータプログラム |
US8890818B2 (en) * | 2010-09-22 | 2014-11-18 | Nokia Corporation | Apparatus and method for proximity based input |
US20120120002A1 (en) * | 2010-11-17 | 2012-05-17 | Sony Corporation | System and method for display proximity based control of a touch screen user interface |
US8686946B2 (en) * | 2011-04-07 | 2014-04-01 | Hewlett-Packard Development Company, L.P. | Dual-mode input device |
WO2013135286A1 (en) * | 2012-03-14 | 2013-09-19 | Nokia Corporation | Touch screen hover input handling |
-
2011
- 2011-10-14 US US13/274,253 patent/US10684768B2/en active Active
-
2012
- 2012-10-12 WO PCT/US2012/059876 patent/WO2013055997A1/en active Application Filing
- 2012-10-12 CN CN201280061711.9A patent/CN104011629B/zh active Active
- 2012-10-12 EP EP12840753.3A patent/EP2766793B1/de active Active
Non-Patent Citations (1)
Title |
---|
None * |
Also Published As
Publication number | Publication date |
---|---|
EP2766793A1 (de) | 2014-08-20 |
EP2766793A4 (de) | 2015-07-29 |
WO2013055997A1 (en) | 2013-04-18 |
US10684768B2 (en) | 2020-06-16 |
CN104011629B (zh) | 2017-12-22 |
CN104011629A (zh) | 2014-08-27 |
US20130097550A1 (en) | 2013-04-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP2766793B1 (de) | Verbesserte zielauswahl für einen berührungsbasierte eingabeaktivierte benutzerschnittstelle | |
US8988366B2 (en) | Multi-touch integrated desktop environment | |
US9600090B2 (en) | Multi-touch integrated desktop environment | |
US9405404B2 (en) | Multi-touch marking menus and directional chording gestures | |
US9501218B2 (en) | Increasing touch and/or hover accuracy on a touch-enabled device | |
US20130061122A1 (en) | Multi-cell selection using touch input | |
TWI451309B (zh) | Touch device and its control method | |
US20110227947A1 (en) | Multi-Touch User Interface Interaction | |
EP3100151B1 (de) | Virtuelle maus für ein gerät mit einem berührungsempfindlichen bildschirm | |
US20140082559A1 (en) | Control area for facilitating user input | |
EP2776905B1 (de) | Interaktionsmodelle für vorrichtungen für indirekte interaktion | |
GB2510333A (en) | Emulating pressure sensitivity on multi-touch devices | |
EP2728456B1 (de) | Verfahren und Vorrichtung zur Steuerung eines virtuellen Bildschirms | |
US9262005B2 (en) | Multi-touch integrated desktop environment | |
EP2661671B1 (de) | Integrierte mehrfachberührungs-desktopumgebung | |
US20150248215A1 (en) | Display of Objects on a Touch Screen and Their Selection | |
US10073612B1 (en) | Fixed cursor input interface for a computer aided design application executing on a touch screen device | |
WO2009119716A1 (ja) | 情報処理システム、情報処理装置、方法及びプログラム | |
WO2017092584A1 (zh) | 一种用于操控操作对象的方法与设备 | |
WO2015029222A1 (ja) | 情報処理装置,表示制御プログラム及び表示制御方法 | |
KR20140086805A (ko) | 전자 장치, 그 제어 방법 및 컴퓨터 판독가능 기록매체 | |
WO2015167531A2 (en) | Cursor grip |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20140514 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
DAX | Request for extension of the european patent (deleted) | ||
RA4 | Supplementary search report drawn up and despatched (corrected) |
Effective date: 20150626 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06F 3/01 20060101AFI20150622BHEP Ipc: G06F 3/0488 20130101ALI20150622BHEP Ipc: G09G 5/00 20060101ALI20150622BHEP |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20170710 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20191202 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602012070258 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1273015 Country of ref document: AT Kind code of ref document: T Effective date: 20200615 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20200520 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200920 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200820 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200821 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200921 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200820 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1273015 Country of ref document: AT Kind code of ref document: T Effective date: 20200520 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602012070258 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
26N | No opposition filed |
Effective date: 20210223 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20201012 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20201031 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20201031 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20201031 Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20201031 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20201031 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20201012 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200520 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 602012070258 Country of ref document: DE Owner name: AUTODESK, INC., SAN FRANCISCO, US Free format text: FORMER OWNER: AUTODESK, INC., SAN RAFAEL, CALIF., US |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230510 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20230920 Year of fee payment: 12 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20230920 Year of fee payment: 12 |