WO2015079527A1 - 情報処理装置、情報処理方法及び情報処理プログラム - Google Patents
情報処理装置、情報処理方法及び情報処理プログラム Download PDFInfo
- Publication number
- WO2015079527A1 WO2015079527A1 PCT/JP2013/082003 JP2013082003W WO2015079527A1 WO 2015079527 A1 WO2015079527 A1 WO 2015079527A1 JP 2013082003 W JP2013082003 W JP 2013082003W WO 2015079527 A1 WO2015079527 A1 WO 2015079527A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- time
- web page
- screen shot
- control unit
- event
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/957—Browsing optimisation, e.g. caching or content distillation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04842—Selection of displayed objects or displayed text elements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/123—Storage facilities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/14—Tree-structured documents
- G06F40/143—Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/0007—Image acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
Definitions
- the present invention relates to a technical field for executing processing corresponding to an element designated by a user among elements constituting a web page.
- Non-Patent Document 1 discloses that when a rendering engine of a browser acquires an HTML document from a network, it starts drawing a web page without waiting for all HTML to be analyzed.
- an operation indicating a link, a button, or the like is performed before rendering of the web page received by the terminal operated by the user is completed.
- the indicated element may be present at a position different from the position at the time of accepting the operation.
- the browser executes the processing by assuming that an element existing at the time of processing of the operation event is indicated at the position pointed by the user. As a result, processing corresponding to an element different from the element intended by the user may be executed.
- the present invention has been made in view of the above points, and even if it takes a long time to render a web page, the processing as intended by the user who performed the operation pointing to the element in the web page is executed. It is an object to provide an information processing apparatus, an information processing method, and an information processing program that can be performed.
- the invention according to claim 1 is configured such that, when an operation indicating the position of an element in the web page is accepted before drawing of the web page is completed, the coordinates indicating the position are An acquisition unit that acquires a screen shot of the web page corresponding to a timing at which the operation is received, and the acquisition unit acquires the screen shot acquired by the acquisition unit.
- the specifying unit for specifying the element indicated by the image at the position indicated by the coordinates and the operation specified for the element specified by the specifying unit are performed when the processing corresponding to the accepted operation event is executed.
- Execution control means for executing a process executed when the event occurs as a process corresponding to the event. And features.
- the information processing apparatus executes processing corresponding to the element indicated by the partial image at the indicated position in the screen shot corresponding to the timing when the operation is received. Therefore, it is possible to execute processing corresponding to the element intended by the user.
- the acquisition unit acquires a screen shot of the web page at every predetermined timing, and the screen acquired by the acquisition unit Image storage control means for associating a shot with the acquisition time of the screen shot and storing them in the storage means, reception time storage control means for storing the reception time of the operation in the storage means, and each of the web pages Completion time storage control means for storing the drawing completion time in the storage means each time drawing of the element is completed, and the completion time older than the reception time among the completion times stored in the storage means If the elapsed time from the specific completion time closest to the reception time to the reception time is less than a reference time, the storage means The storage unit acquires the screenshot corresponding to the acquisition time that is closest to the specific completion time among the acquisition times that are more than the reference time than the specific completion time in the stored acquisition time, and When the elapsed time is equal to or longer than the reference time, the screen corresponding to the acquisition time closest to the specific completion time among
- the process corresponding to the element intended by the user can be executed.
- a first time when drawing of a web page is completed and a second time when an operation indicating the position of an element in the web page is accepted Is the time determined based on the first time and the second time acquired from the history storage means for storing the history as the history, and is the shortest of the time elapsed from the first time to the second time. It further comprises time acquisition means for acquiring time as the reference time.
- an appropriate time can be set as the reference time.
- the second screenshot of the web page is executed when the process corresponding to the accepted operation event is executed.
- the execution control unit When the operation is performed on the element currently displayed at the position indicated by the coordinates acquired by the means, and when there is the image difference, the specifying means is acquired by the acquisition means In the screen shot, the current position of the element indicated by the image at the position indicated by the coordinates acquired by the acquiring means is The execution control means executes the process when the operation is performed on the element currently displayed at the position specified by the specifying means. It is characterized by that.
- the process corresponding to the element intended by the user can be executed by the same process as the conventional one.
- the invention according to claim 5 is an information processing method executed by a computer, and when an operation indicating the position of an element in the web page is accepted before drawing of the web page is completed, the position In the acquisition step of acquiring the coordinates indicating the screen and the screen shot of the web page corresponding to the timing at which the operation is accepted, the acquisition in the acquisition step
- a process executed when the operation is performed is a process corresponding to the event. Characterized in that it comprises a, an execution control step of executing by.
- the computer displays coordinates of the position and the web page.
- a screen shot corresponding to a timing at which the operation is accepted, an acquisition unit that acquires the position, and a position indicated by the coordinates acquired by the acquisition unit in the screen shot acquired by the acquisition unit This is executed when the operation is performed on the element specified by the specifying means when executing the processing corresponding to the event of the accepted operation, and the specifying means for specifying the element indicated by the image in Function as an execution control means for executing the process as a process corresponding to the event. And butterflies.
- the invention according to claim 7 is an information processing program that can be executed simultaneously with a browser by a computer, and that causes the computer to execute an element in the web page before drawing of the web page by the browser is completed.
- An acquisition unit that acquires coordinates indicating the position and a screen shot of the web page corresponding to a timing at which the operation is received, when the operation indicating the position of the operation is received;
- the information processing apparatus executes processing corresponding to the element indicated by the partial image at the indicated position in the screen shot corresponding to the timing when the operation is received. Therefore, it is possible to execute processing corresponding to the element intended by the user.
- FIG. 1 It is a figure which shows an example of schematic structure of the information provision system S which concerns on one Embodiment.
- (A)-(d) is a figure which shows the example of a display of the display area of the web page during rendering.
- (A) is a block diagram which shows an example of schematic structure of the user terminal 1 which concerns on one Embodiment
- (b) is a figure which shows an example of the functional block of the user terminal 1 which concerns on one Embodiment.
- (A) is a flowchart which shows the process example in the point operation reception process of the system control part 16 of the user terminal 1 which concerns on one Embodiment
- (b) is the system control part 16 of the user terminal 1 which concerns on one Embodiment.
- FIG. 6C is a flowchart illustrating a process example in the point operation event control process of the system control unit 16 of the user terminal 1 according to an embodiment. It is a figure which shows an example of schematic structure of the information provision system S which concerns on one Embodiment.
- (A)-(c) is a figure which shows an example of the completion time of drawing of the element of a web page, and the storage time of a screen shot as time passes. It is a figure which shows an example of the functional block of the user terminal 1 which concerns on one Embodiment.
- (A) is a block diagram which shows an example of schematic structure of the information processing server 3,
- (b) is a figure which shows an example of the content registered into log
- FIG. 1 It is a sequence figure showing an example of processing of information processing system S concerning one embodiment.
- A is a flowchart which shows the process example in the storage control process of the system control part 16 of the user terminal 1 which concerns on one Embodiment
- (b) is the system control part 16 of the user terminal 1 which concerns on one Embodiment.
- It is a flowchart which shows the process example in a point operation reception process.
- It is a flowchart which shows the process example in the point operation event control process of the system control part 16 of the user terminal 1 which concerns on one Embodiment.
- (A) and (b) are examples of a screen shot of a web page when there is no image difference
- (c) and (d) are examples of a screen shot of a web page when there is a difference between images. is there.
- (A) is a figure showing an example of a functional block of user terminal 1 concerning one embodiment. It is a flowchart which shows the process example in the point operation event control process of the system control part 16 of the user terminal 1 which concerns on one Embodiment.
- (A) is a flowchart which shows the process example in the point operation reception process of the system control part 16 of the user terminal 1 which concerns on one Embodiment
- (b) is the system control part 16 of the user terminal 1 which concerns on one Embodiment. It is a flowchart which shows the process example in the drawing completion notification reception process of.
- FIG. 1 is a diagram illustrating an example of a schematic configuration of an information providing system S according to the present embodiment.
- the information providing system S includes a plurality of user terminals 1 and a plurality of web servers 2.
- the user terminal 1 and the web server 2 can exchange data with each other via the network NW, for example, using TCP / IP as a communication protocol.
- the network NW is constructed by, for example, the Internet, a dedicated communication line (for example, a CATV (CommunityCommunAntenna Television) line), a mobile communication network (including a base station, etc.), a gateway, and the like.
- the user terminal 1 is a terminal device used by a user who browses a web page.
- the user terminal 1 for example, a personal computer, a PDA (Personal Digital Assistant), a portable information terminal such as a smartphone, a tablet computer, a mobile phone, or the like is used.
- Software such as a browser is incorporated in the user terminal 1.
- the user terminal 1 accesses the web server 2 based on an operation from the user and receives a web page from the web server 2. Then, the user terminal 1 displays the received web page.
- the browser receives data such as an HTML (HyperTextHMarkup Language) document, a CSS (Cascading Style Sheets), an image, and the like from the web server 2.
- the browser constructs a DOM (Document Object Model) tree by analyzing an HTML document, for example, and constructs a render tree based on the DOM tree, CSS, and the like.
- the browser starts rendering before the analysis of the HTML document is completed. The reason is, for example, shortening the time required until the element in the web page is displayed.
- the browser sequentially draws a plurality of elements constituting the web page based on the render tree under construction. Examples of web page elements include links, buttons, characters, images, bars, input forms, tables, lists, and the like. There may be a case where one large element is constituted by a plurality of small elements.
- ⁇ Rendering may take some time from start to completion. Therefore, during the rendering, the user can perform an operation indicating the position of the element displayed in the web page.
- an operation that points to the position of an element is referred to as a point operation.
- a point operation includes a click operation.
- examples of the point operation include a touch operation and a tap operation.
- the browser also processes drawing of each element as an event. This event is called a rendering event.
- This event is called a rendering event.
- the browser adds a rendering event for each element to the event queue while analyzing the HTML document and building a render tree.
- the browser sequentially extracts events from the event queue using FIFO (First In, First First Out), and executes processing corresponding to the extracted events.
- FIFO First In, First First Out
- the browser acquires the coordinates of the position pointed to by the point operation in the display area of the web page. These coordinates are called point coordinates. Then, the browser sets the point coordinates as a parameter for the point operation event, and adds the point operation event to the event queue. Thereafter, when the browser acquires a point operation event from the event queue, the browser specifies the element currently displayed in the display area of the web page at the position indicated by the point coordinates set as a parameter. Then, the browser executes processing (event handler or event listener) corresponding to the point operation event of the identified element.
- processing event handler or event listener
- the elements of the web page are sequentially drawn after the point operation event is received and before the processing corresponding to the point operation event is executed. Therefore, the layout of elements in the display area of the web page may change.
- the element at the position indicated by the point coordinates when the point operation event is received may be different from the element at the position indicated by the point coordinates when the process corresponding to the point operation event is executed.
- the browser executes a process corresponding to the element at the position indicated by the point coordinates when the process corresponding to the point operation event is executed. That is, processing corresponding to an element different from the element pointed by the user is executed.
- the browser when the point operation is accepted, acquires a screen shot of a web page corresponding to the timing at which the point operation is accepted, and stores the screen shot in the memory.
- the screen shot is an image showing the whole or a part of the screen.
- the browser acquires the point coordinates pointed to by the point operation and stores them in the memory. Thereafter, when executing processing corresponding to the point operation event, the browser performs processing corresponding to the element indicated by the portion of the image at the position indicated by the point coordinates stored in the memory in the screen shot stored in the memory. This is executed as processing corresponding to the point operation event.
- FIG. 2A to FIG. 2D are diagrams showing display examples of the display area of the web page during rendering.
- a certain web page is composed of elements CO1 to CO4.
- the element CO1 is drawn.
- the element CO2 is drawn.
- the rendering events for elements CO3 and CO4 have been added to the event queue.
- the user moves the mouse pointer MP by operating the mouse, for example, and clicks at the position of the element CO2.
- the browser acquires a screen shot and point coordinates corresponding to this timing and stores them in the memory. That is, the image of the display area shown in FIG. 2B is stored. Thereafter, as shown in FIG.
- the element CO3 is drawn.
- the position of the element CO2 changes and the element CO2 moves outside the display area.
- the element CO4 is drawn.
- processing corresponding to the point operation event is executed.
- the element CO4 is displayed at the position indicated by the point coordinates.
- the conventional browser executes a process corresponding to the element CO4.
- the browser according to the present embodiment can specify the element CO2 as the element at the position indicated by the point coordinates from the screen shot (FIG. 2B) corresponding to the timing when the point operation is accepted. .
- the browser executes processing corresponding to the element CO2.
- FIG. 3A is a block diagram illustrating an example of a schematic configuration of the user terminal 1 according to the present embodiment.
- the user terminal 1 includes a communication unit 11, a storage unit 12, an operation unit 13, a display unit 14, an input / output interface 15, and a system control unit 16. Yes.
- the system control unit 16 and the input / output interface 15 are connected via a system bus 17.
- the communication unit 11 is connected to the network NW and controls the communication state with the web server 2 and the like.
- the storage unit 12 is configured by, for example, a hard disk drive, a flash memory, or the like.
- the storage unit 12 is an example of a storage unit in the present invention.
- the storage unit 12 stores various programs such as an OS (Operating System) and a browser.
- the browser is an example of an information processing program in the present invention. Note that various programs such as a browser may be acquired from a server device such as the Web server 2 via a network NW, or may be recorded on a recording medium such as an optical disk or a memory card to drive a drive device. It may be configured to be read through.
- the browser may be stored in the storage unit 12 in advance.
- the browser may be a program product.
- the operation unit 13 is an input device that receives an operation from the user such as a point operation.
- the operation unit 13 transmits a signal indicating the operation content from the user to the system control unit 16.
- the operation unit 13 may be configured with at least a keyboard, a mouse, a touch panel, buttons, or the like, for example.
- the display unit 14 is a display device that displays information such as video and characters based on the control of the system control unit 16.
- the display unit 14 may be a liquid crystal display, for example.
- the input / output interface 15 performs interface processing between the communication units 11 to 14 and the system control unit 16.
- the system control unit 16 includes a CPU (Central Processing Unit) 16a, a ROM (Read Only Memory) 16b, a RAM (Random Access Memory) 16c, and the like.
- the RAM 16c is an example of a storage unit in the present invention.
- the CPU 16 is an example of a processor.
- the present invention can also be applied to various processors different from the CPU.
- Each of the storage unit 12, the ROM 16b, and the RAM 16c is an example of a memory.
- the present invention can also be applied to various memories different from the storage unit 12, the ROM 16b, and the RAM 16c.
- FIG. 3B is a diagram illustrating an example of functional blocks of the user terminal 1 according to the present embodiment.
- the system control unit 16 functions as an image storage control unit 161, a coordinate storage control unit 162, an execution control unit 163, and the like as illustrated in FIG. 3B by the CPU 16a executing a program such as a browser.
- the combination of the image storage control unit 161 and the coordinate storage control unit 162 is an example of an acquisition unit in the present invention.
- the coordinate storage control unit 162 is an example of coordinate storage control means in the present invention.
- the execution control unit 163 is an example of a specifying unit and an execution control unit in the present invention.
- the image storage control unit 161 acquires a screen shot of a web page corresponding to the timing when the point operation is accepted. Then, the image storage control unit 161 stores the screen shot in the RAM 16c (or the storage unit 12). For example, when the entire screen of the display unit 14 is a web page display area, the image storage control unit 161 acquires a screen shot of the entire screen. In addition, when a part of the screen of the display unit 14 is a web page display area, the image storage control unit 161 may obtain a screen shot of the entire screen, or a screen shot of only the web page display area. You may get it. When the point operation is accepted, the coordinate storage control unit 162 acquires the point coordinates pointed to by the point operation.
- the coordinate storage control unit 162 stores the point coordinates in the RAM 16c (or the storage unit 12).
- the execution control unit 163 specifies an element indicated by the image of the portion at the position indicated by the point coordinates acquired by the coordinate storage control unit 162 in the screen shot acquired by the image storage control unit 161. Then, the execution control unit 163 causes the process to be executed when the specified element is point-operated as a process corresponding to the accepted point operation event.
- the system control unit 16 adds an event listener for a point operation event.
- the event listener is executed even when the point operation event is received during the rendering based on the HTML document received from the web server 2.
- the system control unit 16 adds an event listener so that the event listener is executed regardless of where the point operation is performed in the display area of the web page. Thereafter, when the drawing of the web page is completed, the system control unit 16 deletes the added event listener.
- FIG. 4A is a flowchart illustrating a processing example in the point operation reception processing of the system control unit 16 of the user terminal 1 according to the present embodiment.
- the image storage control unit 161 acquires a screen shot of the display area of the current web page and stores it in the RAM 16c (step S1).
- the coordinate storage control unit 162 acquires point coordinates, which are one of the parameters of the point operation event, from the point operation event (step S2).
- the coordinate storage control unit 162 adds the special processing event to the event queue secured in the RAM 16c (step S3).
- the special processing event is an event indicating that it is necessary to specify the element pointed to by the user by the point operation. There is a point operation as a parameter of the special processing event.
- the coordinate storage control unit 162 sets the acquired point coordinates as parameters of the special processing event, and adds the special processing event to the event queue.
- step S3 ends, the system control unit 16 ends the point reception process.
- FIG. 4B is a flowchart showing a process example in the event handling process execution control process of the system control unit 16 of the user terminal 1 according to the present embodiment.
- the system control unit 16 starts an event response process execution control process.
- the execution control unit 163 acquires one event from the event queue (step S11).
- the execution control unit 163 determines whether or not the acquired event is a special processing event (step S12).
- step S12 determines that the acquired event is not a special processing event
- the execution control unit 163 proceeds to step S14.
- step S12 determines that the acquired event is a special processing event (step S12: YES)
- step S13 the execution control unit 163 executes a point operation event control process.
- FIG. 4C is a flowchart illustrating a processing example in the point operation event control process of the system control unit 16 of the user terminal 1 according to the present embodiment.
- the execution control unit 163 acquires a screen shot at the time of accepting the point operation from the RAM 16c and also acquires the point coordinates from the special processing event (step S16).
- the execution control unit 163 determines a range of a predetermined number of pixels from the position indicated by the point coordinates as a partial image.
- the execution control unit 163 recognizes one or more characters displayed in the partial image by, for example, known character recognition (step S17). For example, in the example of FIG. 2, “to detailed screen” displayed in the element CO3 is recognized.
- the execution control unit 163 searches the DOM tree for a node that includes the recognized character as text, caption, or the like. Then, the execution control unit 163 acquires the ID of the element corresponding to the searched node (step S18). As a result, the execution control unit 163 identifies the element pointed to by the point operation. Next, the execution control unit 163 changes the event to be processed in the event handling process execution control process from the special event to the point operation event (step S19). Specifically, the execution control unit 163 generates a point operation event. Next, the execution control unit 163 sets the identified element as an event target (step S20). The event target is an element in which a point operation event occurs. Specifically, there is an event target ID as one of the parameters of the point operation event. The execution control unit 163 sets the acquired ID as the event target ID. Further, the execution control unit 163 may set the acquired point coordinates as one of the parameters of the point operation event. When step S20 ends, the system control unit 16 ends the point reception process.
- the execution control unit 163 causes the OS or the like to execute a process corresponding to the acquired event (step S14).
- the execution control unit 163 causes a process corresponding to the point operation event generated by the point operation event control process to be executed.
- the execution control unit 163 may call an API (Application Programming Interface) corresponding to the type of event.
- API Application Programming Interface
- the system control unit 16 on the OS side executes a process corresponding to the event, the event handler of the point operation event of the event target indicated by the ID set as the parameter of the point operation event (or Event listener).
- the system control unit 16 executes a process corresponding to the element indicated by the user by the point operation.
- step S14 the execution control unit 163 determines whether or not the browser is terminated by a user operation (step S15). At this time, if the execution control unit 163 determines that the browser does not end (step S15: NO), the execution control unit 163 proceeds to step S11. On the other hand, if the execution control unit 163 determines that the browser is to be terminated (step S15: YES), the event control process execution control process is terminated.
- the event target ID is not set as the parameter of the point operation event.
- the system control unit 16 on the OS side identifies the element at the position indicated by the point coordinates as the event target at the time of execution of the process corresponding to the point operation event. And the system control part 16 performs the process corresponding to the specified element. Therefore, the problem mentioned above has arisen.
- the system control unit 16 when the point operation is received before the drawing of the web page is completed, corresponds to the point coordinates and the timing at which the point operation is received. Get a screenshot. In addition, the system control unit 16 specifies an element indicated by the partial image at the position indicated by the point coordinates in the acquired screen shot. In addition, when the system control unit 16 executes the process corresponding to the accepted point operation event, the process executed when the point operation is performed on the R identified element corresponds to the point operation event. Run as a process. Therefore, it is possible to execute processing corresponding to the element intended by the user.
- the system control unit 16 performs a process of identifying an element indicated by a part of the image at the position indicated by the point coordinates in the screen shot when the point operation event is executed (event response process execution control process). Steps S17, S18, S20).
- the system control unit 16 may specify an element when accepting a point operation.
- the system control unit 16 sets the ID of the identified element as the event target ID of the point operation event. Then, the system control unit 16 adds the point operation event to the event queue without adding the special processing event.
- the system control unit 16 can execute a process corresponding to the element pointed to by the user by performing a process based on the point operation event added to the event queue.
- FIG. 5 is a diagram illustrating an example of a schematic configuration of the information providing system S according to the present embodiment.
- the information providing system S includes a plurality of user terminals 1, a plurality of web servers 2, and an information processing server 3.
- the user terminal 1, the web server 2, and the information processing server 3 can exchange data with each other via the network NW.
- the information processing server 3 is a web server, for example.
- the information processing server 3 transmits a web page to the user terminal 1 in response to a request from the user terminal 1.
- the configuration of the user terminal 1 is the same as that in the first embodiment.
- reaction time It takes a certain amount of time until the user performs the point operation on the position after the user thinks that the user performs the point operation on the position where the desired element exists. This time is called reaction time.
- the elements that make up the web page are drawn one after another. Therefore, when the point operation is performed on the position where the desired element is present, the desired element may have been moved to another position. Even when the system control unit 16 performs processing using the screen shot stored at the timing when the point operation is received, processing corresponding to an element different from the desired element is executed.
- the system control unit 16 stores screen shots in the RAM 16c at predetermined timings during rendering. And the user terminal 1 determines the screenshot used in order to specify the element pointed out by point operation from the memorize
- FIG. 6A is a diagram illustrating an example of the drawing completion time of the web page element and the storage time of the screen shot as time elapses.
- a web page is composed of contents CO1 to CO4.
- drawing is performed in the order of the contents CO1 to CO4. Specifically, drawing of content CO1 is completed at time FT1, drawing of content CO2 is completed at time FT2, drawing of content CO3 is completed at time FT3, and drawing of content CO4 is completed at time FT4.
- the system control unit 16 stores the screen shot in the RAM 16c, for example, every time a predetermined time elapses after the rendering is started. For example, as shown in FIG.
- screen shots are stored at respective timings of times MT1 to MT5.
- a point operation is accepted at time CT.
- the order of timing is MT4, FT3, CT, MT5, and FT4 in the order of time. Therefore, after the drawing of the element CO3 is completed, the user performs a point operation. Before the element CO3 was drawn, the element CO2 was displayed at the position of the element CO3. Therefore, depending on the point coordinates, it becomes a problem whether the user has performed the point operation in order to designate either the element CO2 or CO3.
- a minimum reaction time when the user performs point operations For example, this time may be set as the reference time.
- a reference time is minimum after drawing of the element is completed.
- the system control unit 16 determines a screen shot for specifying the element pointed to by the point operation based on the elapsed time from the completion time of drawing the element until the point operation is received and the reference time. As shown in FIG. 6B, when the elapsed time from the completion of the drawing of the element CO3 until the point operation is accepted is less than the reference time, the probability that the user thought to point to the element CO3 is low. .
- the system control unit 16 selects the screen shot stored at the latest time MT4 among the screen shots stored before the reference time from the time FT3 among the screen shots before the drawing of the element CO3 is completed.
- the screen shot corresponding to the timing when the point operation is accepted is used to specify the element.
- elements CO1 and CO2 are shown, but element CO3 is not shown.
- FIG. 6C when the elapsed time from the completion of the drawing of the element CO3 until the point operation is accepted is the same as or longer than the reference time, the user receives the element CO3. There is a possibility that he was trying to point to.
- the system control unit 16 uses the screen shot stored at the earliest time MT5 among the screen shots after the drawing of the element CO3 is completed as the screen shot corresponding to the timing at which the point operation is accepted, Used for specific purposes. In this screen shot, elements CO1 to CO3 are shown.
- the reference time may be set in advance.
- the reference time may be determined for each user based on the history of user operations. For example, the time when the drawing of the web page is completed and the time when the point operation is accepted are stored as a history. Then, the shortest time may be determined as the reference time among the time elapsed from the time when the drawing of the web page is completed to the time when the point operation is accepted.
- the information processing server 3 may store the history of all users, and the information processing server 3 may determine the reference time.
- each user terminal may store a history, and each user terminal may determine a reference time. In the present embodiment, the information processing server 3 determines a reference time.
- FIG. 7 is a diagram illustrating an example of functional blocks of the user terminal 1 according to the present embodiment.
- the system control unit 16 functions as an image storage control unit 161, a coordinate storage control unit 162, an execution control unit 163, a time storage control unit 164, a stored image acquisition unit 165, and the like.
- the image storage control unit 161 is an example of image storage control means in the present invention.
- the coordinate storage control unit 162 is an example of a reception time storage control unit in the present invention.
- the time storage control unit 164 is an example of a completion time storage control unit in the present invention.
- the stored image acquisition unit 165 is an example of a stored image acquisition unit and a time acquisition unit in the present invention.
- the image storage control unit 161 stores the screen shot and the storage time of the screen shot in association with each other at a predetermined timing in the RAM 16c.
- the screen shot storage time is also the time when the image storage control unit 161 acquires the screen shot.
- the coordinate storage control unit 162 stores the point coordinates and the point operation acceptance time in the RAM 16c.
- the time storage control unit 164 stores the rendering completion time in the RAM 16c every time rendering of each element constituting the web page is completed.
- the stored image acquisition unit 165 determines the latest completion time among the completion times stored in the RAM 16c older than the point operation reception time during the execution of the process corresponding to the point operation event. Get as.
- the stored image acquisition unit 165 calculates the elapsed time from the specific completion time to the reception time. In addition, when the elapsed time is less than the reference time, the stored image acquisition unit 165 acquires the latest storage time among the storage times stored in the RAM 16c and the storage time older than the specific completion time by the reference time. On the other hand, when the elapsed time is equal to or longer than the reference time, the stored image acquisition unit 165 acquires the oldest storage time among the storage times stored after the specific completion time among the storage times stored in the RAM 16c. Then, the stored image acquisition unit 165 acquires a screen shot corresponding to the acquired storage time from the RAM 16c as a screen shot corresponding to the timing when the point operation is accepted. The execution control unit 163 identifies an event target using the screen shot acquired by the stored image acquisition unit 165.
- FIG. 8A is a block diagram illustrating an example of a schematic configuration of the information processing server 3.
- the information processing server 3 includes a communication unit 31, a storage unit 32, an input / output interface 33, and a system control unit 34.
- the system control unit 34 and the input / output interface 33 are connected via a system bus 35.
- the communication unit 31 is connected to the network NW and controls the communication state with the user terminal 1 and the like.
- the storage unit 32 is constituted by, for example, a hard disk drive.
- the storage unit 32 is an example of a history storage unit in the present invention.
- a database such as a history DB 32a is constructed. “DB” is an abbreviation for database.
- FIG. 8B is a diagram showing an example of contents registered in the history DB 32a.
- a request HTTP (HyperText Transfer Protocol) request
- HTTP HyperText Transfer Protocol
- a history of transmission of a response from the information processing server 3 to the request are registered.
- a user ID, a history type, and a time are associated and registered in the history DB 32a.
- the user ID is user identification information.
- the user ID indicates a user who uses the user terminal 1 that transmitted a request to the information processing server 3 or a user who uses the user terminal 1 that receives a response from the information processing server 3.
- the history type indicates whether the registered history is a response transmission history or a request reception history.
- the response transmission time is registered as the time. If it is a reception history, the reception time of the request is registered as the time.
- the system control unit 34 registers the transmission time of the response as the completion time of drawing the web page by the user terminal 1 that receives the response. Further, the system control unit 34 registers the reception time of the request as the time when the point operation is accepted by the user terminal 1 that transmitted the request.
- the system control unit 34 regards the transmission time of the response as the completion time of drawing.
- the system control unit 34 has a history. May be registered. Such a web page may be determined in advance.
- the transmission time is an example of the first time of the present invention
- the reception time is an example of the second time of the present invention.
- the storage unit 12 also stores various data such as HTML (HyperText Markup Language) documents, XML (Extensible Markup Language) documents, image data, text data, and electronic documents for displaying web pages.
- the storage unit 12 stores various programs such as an OS, a WWW (World Wide Web) server program, and a DBMS (Database Management System).
- the various programs may be acquired from another server device or the like via a network NW, or may be recorded on a recording medium such as an optical disk and read via a drive device.
- the input / output interface 33 performs interface processing between the communication unit 31, the storage unit 32, and the system control unit 34.
- the system control unit 34 includes a CPU 34a, a ROM 34b, a RAM 34c, and the like.
- the system control unit 34 controls each unit of the information processing server 3 by the CPU 34a executing various programs.
- FIG. 9 is a sequence diagram showing a processing example of the information processing system S according to the present embodiment.
- the information processing server 3 transmits an HTML document to the user terminal 1 (step S21).
- the information processing server 3 registers the user ID of the user who uses the user terminal 1 to which the HTML document is transmitted and the transmission time of the HTML document in the history DB 32a (step S22).
- the user terminal 1 that has received the HTML document displays a web page based on the HTML document (step S23). Thereafter, the user performs a point operation, and the user terminal 1 accepts the operation (step S24).
- the user terminal 1 transmits a request corresponding to the element indicated by the point operation to the information processing server 3 (step S25).
- This request includes the user ID of the user who uses the user terminal 1.
- the information processing server 3 that has received the request registers the user ID included in the request and the reception time of the request in the history DB 32a (step S26).
- the information processing server 3 determines a reference time (step S27). Specifically, the information processing server 3 acquires all pairs of transmission time and reception time corresponding to the user ID included in the request from the history DB 32a. Next, the information processing server 3 calculates an elapsed time by subtracting the reception time from the transmission time for each acquired pair.
- the information processing server 3 determines the shortest elapsed time among the calculated elapsed times as the reference time.
- the information processing server 3 transmits the determined reference time to the user terminal 1 (step S28).
- the information processing server 3 may transmit the reference time together with the response to the request, or may transmit the reference time at a predetermined timing.
- the user terminal 1 that has received the reference time stores the reference time in the storage unit 12.
- FIG. 10A is a flowchart illustrating a processing example in the storage control process of the system control unit 16 of the user terminal 1 according to the present embodiment.
- the system control unit 16 executes a storage control process.
- the image storage control unit 161 sets a timer for storing a screen shot (step S31).
- the time storage control unit 164 determines whether or not drawing of any element of the web page has been completed (step S32). At this time, if the time storage control unit 164 determines that the element drawing has been completed (step S32: YES), the process proceeds to step S33.
- step S33 the time storage control unit 164 acquires the current time as the completion time, and stores the completion time in the RAM 16c.
- the time storage control unit 164 proceeds to step S32.
- step S34 the image storage control unit 161 determines whether or not the time set by the timer has elapsed. At this time, if the image storage control unit 161 determines that the set time has elapsed (step S34: YES), the image storage control unit 161 proceeds to step S35. In step S35, the image storage control unit 161 obtains a screen shot of the display area of the current web page. Further, the image storage control unit 161 acquires the current time as the storage time. Then, the image storage control unit 161 stores the acquired screen shot and the storage time in association with each other in the RAM 16c. Next, the image storage control unit 161 proceeds to step S31.
- step S34 determines whether drawing of the web page is completed. At this time, if the time storage control unit 164 determines that the drawing of the web page is not completed (step S36: NO), the process proceeds to step S32. On the other hand, when the time storage control unit 164 determines that the drawing of the web page is completed (step S36: YES), the time storage control unit 164 ends the storage control process.
- FIG. 10B is a flowchart illustrating a processing example in the point operation reception processing of the system control unit 16 of the user terminal 1 according to the present embodiment.
- the coordinate storage unit control 162 acquires the current time as the reception time, and stores the reception time in the RAM 16c (step S4).
- the coordinate storage control unit 162 executes steps S2 and S3 and ends the point operation reception process.
- FIG. 11 is a flowchart illustrating a processing example in the point operation event control process of the system control unit 16 of the user terminal 1 according to the present embodiment.
- the event response process execution control process is the same as that in the first embodiment.
- the stored image acquisition unit 165 acquires a reference time from the storage unit 12 (step S42).
- the stored image acquisition unit 165 acquires the point coordinates from the point operation event, and acquires the point operation reception time from the RAM 16c (step S43).
- the stored image acquisition unit 165 acquires the completion time closest to the reception time among the completion times older than the reception time among the drawing completion times stored in the RAM 16c as the specific completion time (step S44).
- the stored image acquisition unit 165 calculates the elapsed time by subtracting the specific completion time from the reception time (step S45).
- step S46 determines whether or not the elapsed time is less than the reference time. At this time, if the stored image acquisition unit 165 determines that the elapsed time is less than the reference time (step S46: YES), the process proceeds to step S47.
- step S47 the stored image acquisition unit 165 acquires the storage time closest to the specific completion time among the storage times older than the specific completion time among the storage times of the screen shot stored in the RAM 16c. Then, the stored image acquisition unit 165 acquires a screen shot corresponding to the specified storage time from the RAM 16c.
- the stored image acquisition unit 165 proceeds to step S17.
- step S46 determines that the elapsed time is not less than the reference time (step S46: NO)
- the process proceeds to step S48.
- step S48 the stored image acquisition unit 165 acquires the storage time closest to the specific completion time among the storage times after the specific completion time among the storage times of the screen shots stored in the RAM 16c. Then, the stored image acquisition unit 165 acquires a screen shot corresponding to the specified storage time from the RAM 16c.
- step S17 When executing steps S17 to S20, the system control unit 16 ends the point operation event control process.
- the system control unit 16 stores the screen shot of the web page and the storage time in the RAM 16c in association with each other at a predetermined timing. Further, the system control unit 16 stores the point coordinates and the point operation reception time in the RAM 16c. The system control unit 16 stores the drawing completion time in the RAM 16c every time drawing of each element included in the web page is completed. Further, when the elapsed time from the specific completion time to the reception time is less than the reference time, the system control unit 16 corresponds to the storage time closest to the specific completion time among the storage times older than the specific completion time by the reference time or more. A screen shot is acquired from the RAM 16c.
- the system control unit 16 acquires a screen shot corresponding to the storage time closest to the specific completion time from the RAM 16c among the storage times after the specific completion time. And the system control part 16 performs the process corresponding to the element which the partial image in the position which a point coordinate shows in the acquired screen shot. Therefore, even if the position of the element intended by the user changes when the user performs a point operation, the process corresponding to the element intended by the user can be executed.
- system control unit 16 may acquire the shortest time among the time elapsed from the response transmission time to the request reception time as the reference time. In this case, an appropriate time can be set as the reference time.
- the system control unit 16 acquires a screen shot at the time of executing a process corresponding to the point operation event.
- the system control unit 16 determines whether there is an image difference between the screen shot corresponding to the timing at which the point operation is accepted and the screen shot at the time of executing the processing corresponding to the point operation event. For example, the system control unit 16 may determine whether there is a difference in the entire display area of the web page, or may determine whether there is a difference within a predetermined number of pixels in the vertical and horizontal directions from the point coordinates. Good.
- the system control unit 16 performs the same processing as the conventional processing.
- the system control unit 16 specifies the coordinates of the movement destination of the element pointed by the user based on the two screenshots.
- the system control unit 16 sets the identified coordinates as point coordinates that are parameters of the point operation event.
- the system control part 16 performs the process corresponding to a point operation event.
- FIGS. 12A and 12B are examples of screen shots of a web page when there is no image difference.
- elements CO1 and CO4 are displayed as shown in FIG.
- a point operation for the element CO4 is accepted.
- the element CO3 is further drawn.
- the position of the element CO4 has not changed.
- the range for determining the image difference is assumed to be a range R1 that is a predetermined range from the point coordinates.
- the system control unit 16 causes the point operation event to be executed with the point coordinates at the time when the point operation is accepted.
- an element CO4 is displayed at the position indicated by the point coordinates. Therefore, the system control unit 16 executes processing corresponding to the point operation event of the element CO4.
- 12 (c) and 12 (d) are examples of screen shots of a web page when there is a difference between images.
- elements CO1 and CO3 are displayed as shown in FIG.
- a point operation for the element CO3 is accepted.
- the element CO4 is further drawn.
- the element CO3 moved.
- the system control unit 16 specifies the coordinates of the position where the partial image in the range R2 in the screen shot of FIG. 12C has moved in the screen shot of FIG.
- the system control unit 16 causes a point operation event to be executed using the specified coordinates as point coordinates.
- the point coordinates indicate the position where the element CO3 is displayed in FIG. Therefore, the system control unit 16 executes processing corresponding to the point operation event of the element CO3.
- FIG. 13 is a diagram illustrating an example of functional blocks of the user terminal 1 according to the present embodiment.
- the system control unit 16 functions as an image storage control unit 161, a coordinate storage control unit 162, an execution control unit 163, an event execution image acquisition unit 166, a coordinate specification unit 167, and the like.
- the event execution image acquisition unit 166 is an example of an image acquisition unit in the present invention.
- the combination of the coordinate specifying unit 167 is an example of specifying means in the present invention.
- the event execution image acquisition unit 166 acquires a screen shot of the current web page when executing the processing corresponding to the point operation event.
- the coordinate specifying unit 167 corresponds to the timing when the point operation is accepted when there is an image difference between the screen shot corresponding to the timing when the point operation is accepted and the screen shot at the time of executing the process corresponding to the point operation event.
- the current position of the element indicating the partial image at the position indicated by the point coordinates stored in the RAM 16c is specified based on the screen shot at the time of executing the processing corresponding to the point operation event.
- FIG. 14 is a flowchart illustrating a processing example in the point operation event control process of the system control unit 16 of the user terminal 1 according to the present embodiment.
- the point operation event control process is called from the event response process execution control process.
- the event response process execution control process is the same as that in the second embodiment.
- the event execution image acquisition unit 166 acquires a screen shot of the display area of the current web page and stores it in the RAM 16c (step S61).
- the execution control unit 163 acquires a screen shot when the point operation is accepted from the RAM 16c and also acquires the point coordinates from the point operation event (step S62).
- the execution control unit 163 determines whether there is an image difference between the screen shot at the time of accepting the point operation and the current screen shot, for example, by a known background difference method (step S63). At this time, if it is determined that there is no image difference (step S63: NO), the execution control unit 163 ends the point operation event control process.
- step S63 determines that there is an image difference
- step S64 the execution control unit 163 proceeds to step S64.
- step S ⁇ b> 64 the coordinate specifying unit 167 specifies a partial image within the range of a predetermined number of pixels, up and down and left and right of the point coordinates, from the screen shot when the point operation is accepted.
- step S65 the coordinate specifying unit 167 searches for the specified partial image from the screen shot of the current display area by using, for example, a known image matching technique
- step S66 calculates the coordinates of the partial image in the screen shot of the current display area
- the execution control unit 163 changes the event to be processed in the event handling process execution control process from a special event to a point operation event (step S67).
- the execution control unit 163 rewrites the point coordinates of the point operation event with the calculated coordinates (step S698).
- the execution control unit 163 ends the point operation event control process. Since the event target ID is not set as the parameter of the point operation event, in the event handling process execution control process, the system control unit 16 causes the process corresponding to the event currently displayed at the position indicated by the point coordinates to be executed. .
- the execution control unit 163 may not execute the process corresponding to the point operation event, for example.
- the system control unit 16 acquires a screen shot of a web page when executing a process corresponding to a point operation event. Further, when the system control unit 16 has an image difference between the screen shot stored in the RAM 16c and the acquired screen shot, the element indicated by the partial image at the position indicated by the point coordinates in the stored screen shot is displayed. The current position is identified based on the acquired screen shot. And the system control part 16 performs the process corresponding to the element currently displayed in the specified position. On the other hand, when there is no image difference, processing corresponding to the element currently displayed at the position indicated by the point coordinates is executed. Therefore, when there is no image difference, the process corresponding to the element intended by the user can be executed by the same process as before. Further, the system control unit 16 may not search from the DOM tree in order to specify the element.
- the system control unit 16 may acquire a screen shot corresponding to the timing at which the point operation is accepted from among a plurality of screen shots stored in the RAM 16c. Then, the system control unit 16 may determine whether there is an image difference between the acquired screen shot and the screen shot at the time of execution of the process corresponding to the point operation event.
- the information processing program of the present invention is applied to the browser.
- the present embodiment is an embodiment when the information processing program of the present invention is applied to a program (function expansion program) for expanding the function of a browser.
- the function expansion program is a program that the computer can execute simultaneously with the browser.
- the function expansion program is an example of an information processing program in the present invention. Examples of the function expansion program include an add-on program and a toolbar program.
- the user terminal 1 downloads a function expansion program from the web server 2 or the like, stores it in the storage unit 12, and installs the function expansion program.
- the system control unit 16 functions as an image storage control unit 161, a coordinate storage control unit 162, an execution control unit 163, and the like when the CPU 16a executes a function expansion program.
- the function expansion program cannot take out an event from the event queue and control the execution of the process corresponding to the event. Therefore, for example, when the drawing of the web page is completed, it is necessary to notify the function expansion program of the completion of the drawing of the web page from the browser.
- FIG. 15A is a flowchart illustrating a processing example in the point operation reception processing of the system control unit 16 of the user terminal 1 according to the present embodiment.
- the image storage control unit 161 adds an event listener for the point operation event.
- the image storage control unit 161 acquires a screen shot of the display area of the current web page and stores it in the RAM 16c (step S1).
- the execution control unit 163 acquires point coordinates from the point operation event (step S2).
- the coordinate storage control unit 162 stores the acquired point coordinates in the RAM 16c (step S5), and ends the point operation reception process. At this time, the point operation event is not added to the event queue.
- FIG. 15B is a flowchart illustrating a processing example in the drawing completion notification reception process of the system control unit 16 of the user terminal 1 according to the present embodiment.
- the system control unit 16 starts drawing completion notification reception processing.
- the execution control unit 163 determines whether or not a web page drawing completion notification has been received from the browser (step S81). At this time, if the execution control unit 163 determines that a notification has been received (step S81: YES), the execution control unit 163 proceeds to step S82.
- step S82 the execution control unit 163 acquires a screen shot and point coordinates when the point operation is accepted from the RAM 16c.
- the execution control unit 163 determines a range of a predetermined number of pixels as a partial image from the position indicated by the point coordinates in the screen shot. Then, the execution control unit 163 recognizes one or more characters displayed in the partial image (step S83). Next, the execution control unit 163 searches the DOM tree for a node that includes the recognized character as text, caption, or the like. Then, the execution control unit 163 acquires the ID of the element corresponding to the searched node (step S84). Next, the execution control unit 163 calls the event handler of the point operation event of the element indicated by the acquired ID, thereby executing processing corresponding to the point operation event (step S85). Next, the execution control unit 163 proceeds to step S81.
- step S81 when the execution control unit 163 determines that the notification is not received (step S81: NO), the process proceeds to step S86.
- step S86 the execution control unit 163 determines whether or not the browser is terminated by a user operation. At this time, if the execution control unit 163 determines that the browser does not end (step S86: NO), the execution control unit 163 proceeds to step S81. On the other hand, if the execution control unit 163 determines that the browser ends (step S86: YES), the execution control unit 163 ends the drawing completion notification reception process.
- the system control unit 16 receives a web page drawing completion notification from the browser, in the screen shot stored in the RAM 16c, the element indicated by the partial image at the position indicated by the point coordinates is displayed. The corresponding process is executed. Therefore, the function expansion program can also execute processing corresponding to the element intended by the user.
- system control unit 16 may acquire a screen shot corresponding to the timing at which the point operation is accepted from among a plurality of screen shots stored in the RAM 16c.
- the system control unit 16 may acquire a screen shot at the time when receiving a notification of completion of drawing a web page from the browser. Then, the system control unit 16 determines whether or not there is an image difference between the screen shot corresponding to the timing at which the point operation is accepted and the screen shot when the drawing completion notification is received, and based on the determination result. May be processed.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Information Transfer Between Computers (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
[1-1.情報提供システムの構成及び機能概要]
先ず、本実施形態に係る情報提供システムSの構成について、図1及び図2を用いて説明する。図1は、本実施形態に係る情報提供システムSの概要構成の一例を示す図である。
次に、ユーザ端末1の構成について、図3を用いて説明する。
次に、ユーザ端末の動作について、図4を用いて説明する。例えば、ユーザ端末1がウェブサーバ2へHTML文書等のリクエストを送信したとき、システム制御部16は、ポイント操作イベントのイベントリスナーを追加する。これにより、ウェブサーバ2から受信されたHTML文書に基づくレンダリングの最中にポイント操作イベントが受け付けられた場合であっても、イベントリスナーが実行される。また、ウェブページの表示領域内の何れの場所がポイント操作されてもイベントリスナーが実行されるように、システム制御部16はイベントリスナーを追加する。その後、ウェブページの描画が完了したとき、システム制御部16は、追加したイベントリスナーを削除する。
次に、第2実施形態について、図5乃至図12を用いて説明する。
先ず、本実施形態に係る情報提供システムSの構成について、図5を用いて説明する。図5は、本実施形態に係る情報提供システムSの概要構成の一例を示す図である。図5において、図1と同様の要素については同様の符号が付されている。図5に示すように、情報提供システムSは、複数のユーザ端末1と、複数のウェブサーバ2と、情報処理サーバ3とを含んで構成されている。そして、ユーザ端末1とウェブサーバ2と情報処理サーバ3とは、ネットワークNWを介して相互にデータの送受信が可能になっている。情報処理サーバ3は、例えばウェブサーバである。情報処理サーバ3は、ユーザ端末1からの要求に応じて、ウェブページをユーザ端末1へ送信する。なお、ユーザ端末1の構成は第1実施形態の場合と同様である。
図7は、本実施形態に係るユーザ端末1の機能ブロックの一例を示す図である。図7において、図3(b)と同様の要素については同様の符号が付されている。図7に示すように、システム制御部16は、画像記憶制御部161、座標記憶制御部162、実行制御部163、時間記憶制御部164、記憶画像取得部165等としてとして機能する。画像記憶制御部161は、本発明における画像記憶制御手段の一例である。座標記憶制御部162は、本発明における受付時間記憶制御手段の一例である。時間記憶制御部164は、本発明における完了時間記憶制御手段の一例である。記憶画像取得部165は、本発明における記憶画像取得手段及び時間取得手段の一例である。
次に、情報処理サーバ3の構成について、図8を用いて説明する。図8(a)は、情報処理サーバ3の概要構成の一例を示すブロック図である。図8(a)に示すように、情報処理サーバ3は、通信部31と、記憶部32と、入出力インターフェース33と、システム制御部34と、を備えている。そして、システム制御部34と入出力インターフェース33とは、システムバス35を介して接続されている。
次に、情報提供システムSの動作について、図9乃至図11を用いて説明する。
次に、第3実施形態について、図12乃至図14を用いて説明する。なお、情報提供システムSの構成及びユーザ端末1の構成は、第1実施形態の場合と同様である。
本実施形態において、システム制御部16は、ポイント操作イベントに対応する処理の実行時におけるスクリーンショットを取得する。システム制御部16は、ポイント操作が受け付けられたタイミングに対応するスクリーンショットと、ポイント操作イベントに対応する処理の実行時におけるスクリーンショットとで画像の差分があるか否かを判定する。例えば、システム制御部16は、ウェブページの表示領域全体で差分があるかを判定してもよいし、ポイント座標から上下左右、所定ピクセル数の範囲内で、差分があるかを判定してもよい。画像の差分がある場合、システム制御部16は、従来の処理と同様の処理を行う。一方、画像の差分がある場合、システム制御部16は、2つのスクリーンショットに基づいて、ユーザが指した要素の移動先の座標を特定する。システム制御部16は、特定した座標を、ポイント操作イベントのパラメータであるポイント座標として設定する。そして、システム制御部16は、ポイント操作イベントに対応する処理を実行させる。
図13は、本実施形態に係るユーザ端末1の機能ブロックの一例を示す図である。図13において、図3(b)と同様の要素については同様の符号が付されている。図13に示すように、システム制御部16は、画像記憶制御部161、座標記憶制御部162、実行制御部163、イベント実行画像取得部166、座標特定部167等としてとして機能する。イベント実行画像取得部166は、本発明における画像取得手段の一例である。座標特定部167の組合せは、本発明における特定手段の一例である。
次に、ユーザ端末の動作について、図14を用いて説明する。なお、ポイント操作受付処理は、第1実施形態の場合と同様である。
次に、第4実施形態について、図15を用いて説明する。なお、情報提供システムSの構成、ユーザ端末1の構成及びシステム制御部16の機能ブロックは、第1実施形態の場合と同様である。第1~第3実施形態においては、ブラウザに本発明の情報処理プログラムが適用されていた。本実施形態は、ブラウザの機能を拡張するプログラム(機能拡張プログラム)に本発明の情報処理プログラムが適用された場合の実施形態である。機能拡張プログラムは、コンピュータがブラウザと同時に実行することが可能なプログラムである。機能拡張プログラムは、本発明における情報処理プログラムの一例である。機能拡張プログラムとして、例えば、アドオン、ツールバーのプログラム等がある。例えば、ユーザ端末1は、ウェブサーバ2等から機能拡張プログラムをダウンロードして記憶部12に記憶させ、機能拡張プログラムをインストールする。システム制御部16は、CPU16aが、機能拡張プログラムを実行することにより、画像記憶制御部161、座標記憶制御部162、実行制御部163等として機能する。
2 ウェブサーバ
3 情報処理サーバ
11 通信部
12 記憶部
13 操作部
14 表示部
15 入出力インターフェース
16 システム制御部
16a CPU
16b ROM
16c RAM
161 画像記憶制御部
162 座標記憶制御部
163 実行制御部
164 時間記憶制御部
165 記憶画像取得部
166 イベント実行画像取得部
167 座標特定部
17 システムバス
31 通信部
32 記憶部
32a 履歴DB
33 入出力インターフェース
34 システム制御部
34a CPU
34b ROM
34c RAM
35 システムバス
NW ネットワーク
S 情報提供システム
Claims (7)
- ウェブページの描画が完了する前に、前記ウェブページ内の要素の位置を指し示す操作が受け付けられた場合、前記位置を示す座標と、前記ウェブページのスクリーンショットであって、前記操作が受け付けられたタイミングに対応するスクリーンショットと、を取得する取得手段と、
前記取得手段により取得された前記スクリーンショットにおいて、前記取得手段により取得された前記座標が示す位置にある画像が示す要素を特定する特定手段と、
前記受け付けられた操作のイベントに対応する処理の実行時、前記特定手段により特定された前記要素に対して前記操作がされた場合に実行される処理を、前記イベントに対応する処理として実行させる実行制御手段と、
を備えることを特徴とする情報処理装置。 - 請求項1に記載の情報処理装置において、
前記取得手段は、所定のタイミングごとに、前記ウェブページのスクリーンショットを取得し、
前記取得手段により取得された前記スクリーンショットと前記スクリーンショットの取得時間とを対応付けて記憶手段に記憶させる画像記憶制御手段と、
前記操作の受付時間を前記記憶手段に記憶させる受付時間記憶制御手段と、
前記ウェブページに含まれる各要素の描画が完了するごとに、前記描画の完了時間を前記記憶手段に記憶させる完了時間記憶制御手段と、
前記記憶手段に記憶された前記完了時間の中で前記受付時間よりも古い前記完了時間のうち前記受付時間に最も近い特定完了時間から前記受付時間までの経過時間が基準時間未満である場合、前記記憶手段に記憶された前記取得時間の中で前記特定完了時間よりも前記基準時間以上古い前記取得時間のうち前記特定完了時間に最も近い前記取得時間に対応する前記スクリーンショットを前記記憶手段から取得し、前記経過時間が前記基準時間以上である場合、前記記憶手段に記憶された前記取得時間の中で前記特定完了時間以降の前記取得時間のうち前記特定完了時間に最も近い前記取得時間に対応する前記スクリーンショットを前記記憶手段から取得する記憶画像取得手段と、
を更に備え、
前記実行制御手段は、前記記憶画像取得手段により取得された前記スクリーンショットにおいて、前記取得手段により取得された前記座標が示す位置にある画像が示す要素に対応する処理を実行させることを特徴とする情報処理装置。 - 請求項2に記載の情報処理装置において、
ウェブページの描画が完了した第1時間と、前記ウェブページ内の要素の位置を指し示す操作が受け付けられた第2時間とを履歴として記憶する履歴記憶手段から取得された前記第1時間及び前記第2時間に基づいて決定された時間であって、前記第1時間から前記第2時間まで経過した時間のうち最短の時間を前記基準時間として取得する時間取得手段を更に備えることを特徴とする情報処理装置。 - 請求項1乃至3の何れか1項に記載の情報処理装置において、
前記受け付けられた操作のイベントに対応する処理の実行時、前記ウェブページの第2スクリーンショットを取得する画像取得手段と、
前記取得手段により取得された前記スクリーンショットと前記画像取得手段により取得された前記第2スクリーンショットとの間で画像差分がない場合、前記実行制御手段は、前記取得手段により取得された前記座標が示す位置に現在表示されている要素に対して前記操作がされた場合の処理を実行させ、
前記画像差分がある場合、前記特定手段は、前記取得手段により取得されたスクリーンショットにおいて前記取得手段により取得された前記座標が示す位置にある画像が示す要素の現在位置を、前記画像取得手段により取得された第2スクリーンショットに基づいて特定し、前記実行制御手段は、前記特定手段により特定された前記位置に現在表示されている要素に対して前記操作がされた場合の処理を実行させることを特徴とする情報処理装置。 - コンピュータにより実行される情報処理方法であって、
ウェブページの描画が完了する前に、前記ウェブページ内の要素の位置を指し示す操作が受け付けられた場合、前記位置を示す座標と、前記ウェブページのスクリーンショットであって、前記操作が受け付けられたタイミングに対応するスクリーンショットと、を取得する取得ステップと、
前記取得ステップにより取得された前記スクリーンショットにおいて、前記取得ステップにより取得された前記座標が示す位置にある画像が示す要素を特定する特定ステップと、
前記受け付けられた操作のイベントに対応する処理の実行時、前記特定ステップにより特定された前記要素に対して前記操作がされた場合に実行される処理を、前記イベントに対応する処理として実行させる実行制御ステップと、
を含むことを特徴とする情報処理方法。 - コンピュータを、
ウェブページの描画が完了する前に、前記ウェブページ内の要素の位置を指し示す操作が受け付けられた場合、前記位置を示す座標と、前記ウェブページのスクリーンショットであって、前記操作が受け付けられたタイミングに対応するスクリーンショットと、を取得する取得手段、
前記取得手段により取得された前記スクリーンショットにおいて、前記取得手段により取得された前記座標が示す位置にある画像が示す要素を特定する特定手段、及び、
前記受け付けられた操作のイベントに対応する処理の実行時、前記特定手段により特定された前記要素に対して前記操作がされた場合に実行される処理を、前記イベントに対応する処理として実行させる実行制御手段、
として機能させることを特徴とする情報処理プログラム。 - コンピュータがブラウザと同時に実行することが可能な情報処理プログラムであって、
前記コンピュータを、
前記ブラウザによるウェブページの描画が完了する前に、前記ウェブページ内の要素の位置を指し示す操作が受け付けられた場合、前記位置を示す座標と、前記ウェブページのスクリーンショットであって、前記操作が受け付けられたタイミングに対応するスクリーンショットと、を取得する取得手段、
前記取得手段により取得された前記スクリーンショットにおいて、前記取得手段により取得された前記座標が示す位置にある画像が示す要素を特定する特定手段、
前記ウェブページの描画の完了の通知を前記ブラウザから受信する受信手段、及び、
前記受信手段により前記通知が受信されたとき、前記特定手段により特定された前記要素に対して前記操作がされた場合に実行される処理を実行させる実行制御手段、
として機能させることを特徴とする情報処理プログラム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/JP2013/082003 WO2015079527A1 (ja) | 2013-11-28 | 2013-11-28 | 情報処理装置、情報処理方法及び情報処理プログラム |
JP2014512999A JP5612796B1 (ja) | 2013-11-28 | 2013-11-28 | 情報処理装置、情報処理方法及び情報処理プログラム |
US14/407,122 US9875316B2 (en) | 2013-11-28 | 2013-11-28 | Identifying user selection using coordinates and snapshots of webpages |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/JP2013/082003 WO2015079527A1 (ja) | 2013-11-28 | 2013-11-28 | 情報処理装置、情報処理方法及び情報処理プログラム |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2015079527A1 true WO2015079527A1 (ja) | 2015-06-04 |
Family
ID=52574656
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2013/082003 WO2015079527A1 (ja) | 2013-11-28 | 2013-11-28 | 情報処理装置、情報処理方法及び情報処理プログラム |
Country Status (3)
Country | Link |
---|---|
US (1) | US9875316B2 (ja) |
JP (1) | JP5612796B1 (ja) |
WO (1) | WO2015079527A1 (ja) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2016154051A (ja) * | 2016-05-20 | 2016-08-25 | カシオ計算機株式会社 | コンテンツ出力装置及びプログラム |
US9886403B2 (en) | 2014-04-03 | 2018-02-06 | Casio Computer Co., Ltd. | Content output device for outputting content |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10795547B1 (en) * | 2014-06-11 | 2020-10-06 | Amazon Technologies, Inc. | User-visible touch event queuing |
CN111090375B (zh) * | 2019-11-21 | 2023-07-25 | 深圳前海环融联易信息科技服务有限公司 | 一种智能网页截图方法、装置、计算机设备及存储介质 |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2011065490A (ja) * | 2009-09-17 | 2011-03-31 | Sony Corp | 情報処理装置、データ取得方法及びプログラム |
JP2011164855A (ja) * | 2010-02-08 | 2011-08-25 | Fujitsu Toshiba Mobile Communications Ltd | 情報処理装置 |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6978418B1 (en) * | 1999-04-13 | 2005-12-20 | Hewlett-Packard Development Company, L.P. | Dynamic-adaptive client-side image map |
-
2013
- 2013-11-28 WO PCT/JP2013/082003 patent/WO2015079527A1/ja active Application Filing
- 2013-11-28 US US14/407,122 patent/US9875316B2/en active Active
- 2013-11-28 JP JP2014512999A patent/JP5612796B1/ja active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2011065490A (ja) * | 2009-09-17 | 2011-03-31 | Sony Corp | 情報処理装置、データ取得方法及びプログラム |
JP2011164855A (ja) * | 2010-02-08 | 2011-08-25 | Fujitsu Toshiba Mobile Communications Ltd | 情報処理装置 |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9886403B2 (en) | 2014-04-03 | 2018-02-06 | Casio Computer Co., Ltd. | Content output device for outputting content |
JP2016154051A (ja) * | 2016-05-20 | 2016-08-25 | カシオ計算機株式会社 | コンテンツ出力装置及びプログラム |
Also Published As
Publication number | Publication date |
---|---|
US9875316B2 (en) | 2018-01-23 |
JPWO2015079527A1 (ja) | 2017-03-16 |
US20160275200A1 (en) | 2016-09-22 |
JP5612796B1 (ja) | 2014-10-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3518124A1 (en) | Webpage rendering method and related device | |
KR101651524B1 (ko) | 브라우저의 새로운 탭 페이지 및 북마크 툴바 | |
CN106844635B (zh) | 网页中的元素的编辑方法及装置 | |
WO2017215175A1 (zh) | 页面处理方法、装置、终端及服务器 | |
US20070067333A1 (en) | Web browsing method and system, and recording medium thereof | |
US9477762B2 (en) | Search controls using sliders and lightboxes | |
JP5612796B1 (ja) | 情報処理装置、情報処理方法及び情報処理プログラム | |
US20170169122A1 (en) | Webpage display method, mobile terminal, intelligent terminal, program and storage medium | |
WO2015035897A1 (en) | Search methods, servers, and systems | |
US20160004676A1 (en) | Displaying web pages | |
CN107451163B (zh) | 一种动画显示方法和装置 | |
JP5437325B2 (ja) | 情報処理装置、システム、方法及びプログラム | |
WO2016011699A1 (zh) | 一种用于配置浏览器的导航页的方法与装置 | |
KR101507779B1 (ko) | 모션 인식을 이용한 즐겨찾기 처리 방법 | |
JP7206863B2 (ja) | 表示制御方法、装置、及びプログラム | |
JP2012173946A (ja) | ウェブページ表示装置およびフォーカス制御方法 | |
JP2012181693A (ja) | ウェブページ表示制御装置およびスクロール制御方法 | |
JP5538459B2 (ja) | 情報処理装置及び方法 | |
KR20110129141A (ko) | 단말 기기에서 하나의 웹 사이트를 통해 복수 개의 웹 사이트를 검색하는 장치 및 방법 | |
KR20100079052A (ko) | 캐쉬(cache)를 이용하여 웹 페이지를 제공하기 위한 방법, 시스템 및 컴퓨터 판독 가능한 기록 매체 | |
JP2007323554A (ja) | ネットワーク検索方法,クライアント,サーバ,システム | |
WO2018154801A1 (ja) | ウェブページ制作支援システム | |
JP2009015378A (ja) | ページ閲覧装置 | |
US11120207B2 (en) | Information processing apparatus, information processing method, and storage medium | |
JP2013020459A (ja) | 翻訳装置及び方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
ENP | Entry into the national phase |
Ref document number: 2014512999 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14407122 Country of ref document: US |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 13898158 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 13898158 Country of ref document: EP Kind code of ref document: A1 |