WO2020022780A1 - Method and apparatus for establishing device connection - Google Patents
Method and apparatus for establishing device connection Download PDFInfo
- Publication number
- WO2020022780A1 WO2020022780A1 PCT/KR2019/009198 KR2019009198W WO2020022780A1 WO 2020022780 A1 WO2020022780 A1 WO 2020022780A1 KR 2019009198 W KR2019009198 W KR 2019009198W WO 2020022780 A1 WO2020022780 A1 WO 2020022780A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- iot
- image
- list
- request
- server
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
- G06F1/16—Constructional details or arrangements
- G06F1/1613—Constructional details or arrangements for portable computers
- G06F1/163—Wearable computers, e.g. on a belt
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/40—Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/0482—Interaction with lists of selectable items, e.g. menus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04886—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/17—Image acquisition using hand-held instruments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/94—Hardware or software architectures specially adapted for image or video understanding
- G06V10/95—Hardware or software architectures specially adapted for image or video understanding structured as a network, e.g. client-server architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0806—Configuration setting for initial configuration or provisioning, e.g. plug-and-play
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/085—Retrieval of network configuration; Tracking network configuration history
- H04L41/0853—Retrieval of network configuration; Tracking network configuration history by actively collecting configuration information or by backing up configuration information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/22—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks comprising specially adapted graphical user interfaces [GUI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L61/00—Network arrangements, protocols or services for addressing or naming
- H04L61/50—Address allocation
- H04L61/5038—Address allocation for local use, e.g. in LAN or USB networks, or in a controller area network [CAN]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- H04L67/125—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks involving control of end-device applications over a network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/80—Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W76/00—Connection management
- H04W76/10—Connection setup
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W76/00—Connection management
- H04W76/10—Connection setup
- H04W76/11—Allocation or use of connection identifiers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W8/00—Network data management
- H04W8/005—Discovery of network devices, e.g. terminals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L61/00—Network arrangements, protocols or services for addressing or naming
- H04L61/45—Network directories; Name-to-address mapping
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/724—User interfaces specially adapted for cordless or mobile telephones
- H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
- H04M1/72409—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality by interfacing with external accessories
- H04M1/72415—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality by interfacing with external accessories for remote control of appliances
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M2250/00—Details of telephonic subscriber devices
- H04M2250/52—Details of telephonic subscriber devices including functional features of a camera
Definitions
- the disclosure relates to the technical field of communication. Particularly, the disclosure relates to a method and apparatus for establishing a device connection.
- a current user needs to open the Bluetooth or a wireless network first, search for a device list in the current IoT environment, select the target device in the list, try to establish a connection, and enter a connection password until the connection is successfully established.
- IO input/output
- an aspect of the disclosure is to provide an apparatus and method for establishing a device connection, which can automatically establish a connection between devices with less operation on the devices.
- a method for establishing device connections, applied to a first device with a camera includes acquiring an image of a second device by the first device through the camera, recognizing the type of the second device through the acquired image of the second device, acquiring a device list corresponding to the type of the second device through an Internet-of-things server, and identifying the second device in the device list, acquiring a unique identifier of the second device in the current Internet-of-things environment, and using the unique identifier of the second device to initiate a connection request to the second device to establish a connection with the second device.
- a device for establishing a device connection applied to a first device includes a camera includes a first acquisition unit, a recognition unit, a second acquisition unit, a determination unit, a third acquisition unit, and a processing unit, in which the first acquisition unit is configured to acquire an image of the second device through the camera, the recognition unit is configured to recognize the type of the second device through the image of the second device acquired by the first acquisition unit, the second acquisition unit is configured to acquire, through an Internet-of-things server, a device list corresponding to the type of the second device recognized by the recognition unit, the determination unit is configured to identify the second device in the device list acquired by the second acquisition unit, the third acquisition unit is configured to acquire a unique identifier of the second device in the current Internet-of-things environment determined by the determination unit, and the processing unit is configured to use the unique identifier of the second device acquired by the third acquisition unit to initiate a connection request to the second device to establish a connection with the second device.
- the first acquisition unit is configured to acquire an image of the second device
- the first device captures, through the camera, and recognizes the device type of the second device to be connected, and acquires the device list corresponding to the type of the second device from the Internet-of-things server to determine the related information of the second device so as to establish a connection with the second device. Establishing a connection between devices with less operation on the devices can be implemented in this solution.
- an aspect of the disclosure is to provide an apparatus and method for establishing a device connection, which can automatically establish a connection between devices with less operation on the devices.
- FIG. 1 is a flowchart of establishing a device connection according to an embodiment of the disclosure
- FIG. 2 is a diagram of a first device acquiring an image of a second device through a camera according to an embodiment of the disclosure
- FIG. 3 is a flowchart of a first device controlling a second device through an internet of things (IoT) menu according to an embodiment of the disclosure
- FIG. 4 is a diagram of initiating a request for an IoT menu by clicking according to an embodiment of the disclosure
- FIG. 5 is a diagram of displaying an IoT menu of a second device on a visual floating window on a first device according to an embodiment of the disclosure
- FIG. 6 is a diagram of selecting a mute function in an IoT menu of the second device according to an embodiment of the disclosure
- FIG. 7 is a flowchart of establishing a device connection according to an embodiment of the disclosure.
- FIG. 8 is a diagram of establishing a device connection with a pair of glasses according to an embodiment of the disclosure.
- FIG. 9 is a flowchart of displaying an IoT state of devices according to an embodiment of the disclosure.
- FIG. 10 is a diagram of displaying an unconnected state according to an embodiment of the disclosure.
- FIG. 11 is a diagram of displaying a connected state in according to an embodiment of the disclosure.
- FIG. 12 is a diagram of the structure of an apparatus according to an embodiment of the disclosure.
- FIG. 13 is a block diagram of a device in a network environment according to an embodiment of the disclosure.
- An embodiment of the disclosure provides a method for establishing a device connection which applied to a first device with a camera.
- the first device captures and recognizes the device type of a second device to be connected through the camera, and acquires a device list corresponding to the type of the second device from an internet of things (IoT) server to determine the related information of the second device so as to establish a connection with the second device.
- IoT internet of things
- the devices such as the first device, the second device and the like are not specifically referred to as a type of devices, but refer to any device in the current IoT environment.
- the first device is taken as a main body for controlling the connection establishment.
- the first device may be a mobile phone, an iPad®, glasses or the like; and the first device needs to have camera function, that is, with a camera.
- a process of one device establishing a connection with another device is taken as an example.
- FIG. 1 is a flowchart of establishing a device connection in according to an embodiment of the disclosure.
- a first device 200 acquires an image of a second device 210 through a camera (e.g., a camera module).
- a camera e.g., a camera module
- the first device 100 displays the image on a screen (e.g., a touch screen display). That is, after the first device captures the image of the second device, all the captured contents are displayed on the screen.
- a screen e.g., a touch screen display
- a command for selecting any one device may include various types of commands such as a touch (e.g., tap) gesture, a swipe gesture, a drag gesture, a hovering gesture, or a voice command.
- multiple devices when the target image is captured, multiple devices may be relatively close to each other, and the multiple devices may be captured at the same time. If the first device is moved or the user moves, there will be some inconvenience, or the second device 210 may not be accurately captured. In the disclosure, when this situation occurs, the user may manually select the target device, i.e., the second device, by touch, such that the first device 200 will know which device is to be recognized later. And if the first device 200 is a pair of glasses (e.g., Google glasses), instructions may be input through some virtual gestures or body movements.
- a pair of glasses e.g., Google glasses
- the first device 200 recognizes a type of the second device 210 through the acquired image of the second device 210.
- the first device 200 acquires a device list corresponding to the type of the second device 210 through the IoT server. After the device list is acquired, operation 103 is performed. And if the device list corresponding to the type of the second device 210 is not acquired through the IoT server, the IoT server is notified to establish a device list corresponding to the type of the second device 210, and add the feature information corresponding to the second device 210 in the established device list.
- notifying the second device 210 by the first device 200 to add the device list and the feature information may be implemented by connecting the first device 200 to the IoT server or may be implemented by notifying the IoT server to manually add the first device 200 at the IoT server.
- the implementation of adding a device list corresponding to one device type to the IoT server is not limited in the disclosure.
- a device list of refrigerators is added to the IoT server.
- the function or operation of adding the device list of refrigerators may include, for example, assigning a unique identifier to the second device 210, and storing the brand of the device, the current state of the device (power-on, standby, contents being played, and the like), the name of the device, the model of the device, the surrounding environment of the device and the like.
- the feature information of the device may be added manually or by voice input, or may be added by recognizing the image of the second device 210 to acquire the feature information.
- FIG. 2 is a schematic diagram of a first device acquiring an image of a second device through a camera according to an embodiment of the disclosure.
- the example illustrates that the first device 200 is a mobile terminal and the second device 210 is a television set.
- the first device 200 may also be a pair of glasses with a camera.
- the first device 200 captures an image of the second device 210 through the camera.
- the device type of the second device 210 is recognized in the back-end (e.g., a server), and the second device is further identified in the corresponding device list.
- the information about the second device is identified and displayed on the captured image in a preset mode (e.g., in a form of a dashed box on the contour boundary of the second device).
- the information may be not displayed, and the second device is directly determined in the back-end.
- the first device 200 acquires a device list corresponding to a type of the second device 210 through a server (e.g., the IoT server) and identifies the second device 210 in the device list.
- a server e.g., the IoT server
- the feature information of the recognized device should match and identify the second device 210.
- the feature information of the second device 210 is recognized according to the acquired image of the second device 210, which may be specifically implemented by determining the feature information of the second device 210 based on preset recognition and some local pre-configuration.
- the method for recognizing the feature information of the second device 210 is not limited in the disclosure. The method may be implemented according to the existing implementation. And then, the second device 210 is determined in the device list corresponding to the type of the second device 210 acquired through the IoT server, which is described as below.
- the feature information of the second device 210 is used to match with the feature information corresponding to each unique identifier in the device list. That is, the feature information corresponding to each unique identifier is matched. And if there is the feature information that matches the feature information of the second device 210, the device identified by the unique identifier corresponding to the matched feature information is determined as the second device 210.
- matching rules may be preset to determine whether the feature information matches, and the following rules may be followed but are not limited thereto.
- each piece of feature information of any one device is separately matched separately, and a device of which the number of matched feature information reaches the preset number serves as a device which is matched with the feature information of the second device 210, and serves as the target device.
- a device of which the number of matched feature information reaches the preset number serves as a device which is matched with the feature information of the second device 210, and serves as the target device.
- the preset value is set to 3
- the device with 3 or more matched feature information will serve as the target device, that is, the second device 210 to be determined.
- the device with the most matched feature information is selected as the target device; or the device of which the feature information matches certain preset feature information is determined as the target device.
- the devices in the device list are the same type (e.g., the same refrigerator) as the second device 210 but are not the second device 210, none of the multiple refrigerators in the refrigerator list is the refrigerator indicated by the second device 210.
- the first device 200 notifies the IoT server to assign a unique identifier to the second device 210 in the device list, and add the corresponding feature information.
- the first device 200 may send the feature information of the second device 210 to the IoT server, and the IoT server may add the feature information of the second device 210 to the device list corresponding to the second device 210 and assign the unique identifier.
- the IoT server may be only notified that the new device is not in the device list, and the related information may be directly (e.g., manually) added in the device list on the IoT server side.
- the first device 200 acquires a unique identifier of the second device in the current IoT environment (e.g., from the IoT server).
- the first device 200 uses the unique identifier of the second device 210 to transmit (e.g., initiate) a connection request to the second device 210 to establish a connection with the second device 210.
- the first device 200 and the second device 210 negotiate and select an IoT protocol supported by the two devices, and establish a connection by interaction.
- the process of negotiating and establishing a connection is similar to the existing implementation, which will not be described here in detail.
- a device with a camera may automatically establish a connection with the target device. Manually searching all the IoT devices in the current environment and manually selecting the target device and establishing a connection are not needed in this embodiment.
- This embodiment describes the process of controlling one device through another device after a connection has been established between the two devices.
- FIG. 3 is a flowchart of a first device controlling a second device through an IoT menu according to an embodiment of the disclosure.
- the first device 200 after receiving a request input by a user for acquiring an IoT menu 500 of the second device 210, the first device 200 sends, to the second device 210, a request for acquiring the IoT menu of the second device 210.
- the method for sending a request for acquiring the IoT menu 500 may be implemented by a menu request that is triggered by preconfigured related operations. For example, if the picture of the second device is touched by clicking or sliding on the screen of the first device, the first device receives the request for the IoT menu 500 input by the user.
- FIG. 4 is a diagram of initiating a request for an IoT menu by clicking according to an embodiment of the disclosure.
- At least one of the first device or the second device determines that a request is input by the user for acquiring an IoT menu 500 of the second device. Then a request for acquiring the IoT menu 500 of the second device is sent to the second device.
- the IoT menu 500 herein may include a function menu of the second device.
- the IoT menu 500 may include at least one menu among a menu 501 for turning on/off power, menus 502 and 503 for controlling volume, menus 504 and 505 for switching channels, or a menu 506 for muting.
- the request may also be input by voice.
- the second device 210 in response to a request for an IoT menu sent by the first device 200, sends the data related to the IoT menu to the first device 200.
- the first device 200 displays information about the IoT menu (e.g., a menu related to a volume control function, a menu related to channel switch function, a menu related to a device power on/off function, or a menu related to a muting function) in a visual floating window.
- information about the IoT menu e.g., a menu related to a volume control function, a menu related to channel switch function, a menu related to a device power on/off function, or a menu related to a muting function
- FIG. 5 is a diagram of displaying an IoT menu of a second device on a visual floating window on a first device according to an embodiment of the disclosure.
- the contour boundary of the image of the second device 210 displayed on the first device 200 may be recognized, and the IoT menu of the second device is preferably displayed outside the contour boundary of the image of the second device, thus not affecting the viewing of the image of the second device.
- the first device after receiving a request by the user for selecting any menu option in the IoT menu, the first device sends the request to the second device such that the second device performs related operations according to the request.
- any menu option may also be selected by voice or touch. If the first device is a pair of glasses, the touch control may be performed through virtual gestures and the like.
- FIG. 6 is a diagram of selecting a mute menu in an IoT menu of a second device according to an embodiment of the disclosure.
- the first device 200 e.g., a mobile phone
- the second device 210 e.g., a television set
- the second device 210 mutes the television set as if it received a request for "mute” function from a remote controller.
- the first device 200 may control the second device 210 to perform the functional operation of the second device 210.
- the user may control the second device 210 in different places in a room, such as controlling a speaker to play music.
- a process of at least one device assisting other devices to establish a connection is taken as an example.
- FIG. 7 is a flowchart of establishing a connection between devices according to an embodiment of the disclosure.
- the first device e.g., the first device 200 of FIG. 1 recognizes a type of a third device 800 through a camera.
- the first device 200 recognizes a type of the third device 800 through the acquired image.
- the first device 200 acquires a device list corresponding to the type of the third device 800 through an IoT server and identifies the third device 800 in the device list.
- the first device 200 captures an image of a fourth device 810 through the camera.
- the first device 200 recognizes a type of the fourth device 810 through the acquired image, acquires a device list corresponding to the type of the fourth device 810 through an IoT server, and identifies the fourth device 810 in the device list.
- operation 701 and operation 702 are in no particular order.
- the specific implementation of operation 701 and operation 702 may refer to the specific implementation of operations 101 to 103.
- the first device 200 in this example is a pair of glasses.
- FIG. 8 is a diagram of establishing a connection between devices with a pair of glasses according to an embodiment of the disclosure.
- the process of acquiring an image with glasses is similar to the implementation of a mobile terminal, except that the image or menu is displayed through a virtual screen.
- the input of instructions may be implemented through virtual gestures or voice.
- the process of identifying the third device 800 and the fourth device 810 in the device list in this embodiment is similar to the process of identifying the second device 210 in the first embodiment.
- the first device 200 determines whether the fourth device 810 initiates a request for establishing connection. If the fourth device 810 initiates the request for establishing connection, operation 704 is performed. Otherwise, operation 705 is performed.
- the first device 200 acquires the device information of the third device 800 and sends the device information to the fourth device 810.
- the fourth device 810 uses the device information, the fourth device 810 initiates a connection request for establishing a connection to the third device 800 and establishes a connection with the third device 800.
- the first device 200 acquires the device information of the fourth device 810 and sends the device information to the third device 800.
- the third device 800 uses the device information, the third device 800 initiates a connection request for establishing a connection to the fourth device 810 and establishes a connection with the fourth device 810.
- the device information of the third device 800 or the fourth device 810 which are acquired by the first device 200, may be a unique identifier of the third device 800 or the fourth device 810 in the current IoT environment, such that the device that initiates the establishment of connection may initiate a request to the peer device.
- the third device 800 or the fourth device 810 acquires the related information of the peer device.
- the process of connection is similar to the existing implementation.
- the user is not required to manually select the target device in the list and is very convenient to establish a connection between devices without a display or with a display that is too small.
- the fourth embodiment provides a process of viewing the state of the IoT device.
- FIG. 9 is a flowchart of displaying an IoT state of devices according to an embodiment of the disclosure.
- the first device 200 captures an image of a fifth device through the camera.
- the first device 200 recognizes a type of the fifth device through the acquired image.
- the first device 200 acquires a device list corresponding to the type of the fifth device through the IoT server and identifies the fifth device in the device list.
- the implementation process of operations 901 to 903 is similar to the implementation process of operations 101 to 103, which will not be described here in detail.
- the first device 200 acquires an IoT state of the identified fifth device through the IoT server.
- the other device After each device is connected to another device, the other device is recorded in the IoT server. And the other device may be recorded in a device list or may be separately stored.
- the first device 200 outputs the IoT state of the fifth device through a visual floating window or auditory effects.
- the IoT states of the device include connected and unconnected states.
- the unconnected state may not be identified, or may be identified by a text, or may also be identified by a certain predetermined shape, such as a rectangular frame, a circle or the like.
- the unconnected state when the unconnected state is displayed, it may be displayed without limitation, or may be displayed in such a manner as not to block the position of the image of the fifth device.
- the unconnected state is displayed at a position other than the image of the fifth device.
- FIG. 10 is a diagram of displaying an unconnected state according to an embodiment of the disclosure.
- an unconnected state is identified with a dashed box displayed outside the contour boundary, but various embodiments of the disclosure are not limited thereto.
- a connected state may be identified by text or may be identified by a certain predetermined shape, and only needs to be distinguished from the unconnected state.
- the connected state when the connected state is displayed, it may be displayed without limited, or may be limited to be displayed at a position outside the image of the fifth device, in such a manner as not to block the fifth device.
- a list of the devices connected to the fifth device is also acquired through the IoT server, and displayed in the form of a floating menu at a position other than the position of the image of the fifth device.
- the connected device may be displayed by name when displayed, or the unique identifier of the device may be displayed, for the purpose that the user may sure the connected devices.
- FIG. 11 is a diagram of displaying a connected state according to an embodiment of the disclosure.
- a connected state is identified with a solid box displayed outside the contour boundary, but various embodiments of the disclosure are not limited thereto.
- the device list 1110 of the connected devices specifically including Samsung® Galaxy TM S8+, Samsung® Gear2, Device2 and unnamed, is displayed outside the image of the fifth device.
- connected devices mean devices that are currently in a connected state or have ever been connected.
- the first device 200 with a camera captures and recognizes other smart devices (the second device 210, the third device 800, the fourth device 810, and the fifth device 1100) in the current IoT environment through the camera, such that the first device 200 may quickly establish a connection with the devices (the second device 210, the third device 800, the fourth device 810, and the fifth device 1100); or quickly establish a connection between the devices (the second device 210, the third device 800, the fourth device 810, and the fifth device 1100).
- the state of the devices (the second device 210, the third device 800, the fourth device 810, and the fifth device 1100) and an operation menu are displayed on the picture captured through the camera of the first device in augmented reality.
- the operation intervention by a user during the interconnection of devices is minimized in the disclosure.
- the intention of the user is intelligently recognized and automatically executed (the establishment of a connection), and a user interface (UI) operation interface of augmented reality and the identification information of devices are provided.
- UI user interface
- the disclosure further proposes an apparatus for establishing device connection, which is implemented by the first device 200 with a camera.
- FIG. 12 is a diagram of an apparatus according to an embodiment of the disclosure.
- the apparatus includes a first acquisition unit 1201, a recognition unit 1202, a second acquisition unit 1203, a determination unit 1204, a third acquisition unit 1205 and a processing unit 1206. At least one operation performed by the units 1201 through 1206 according to various embodiments of the disclosure may be performed by at least one processor 1320.
- the first acquisition unit 1201 is configured to acquire an image of a second device through the camera.
- the recognition unit 1202 is configured to recognize the type of the second device through the image of the second device acquired by the first acquisition unit 1201.
- the second acquisition unit 1203 is configured to acquire, through an IoT server, a device list corresponding to the type of the second device recognized by the recognition unit 1202.
- the determination unit 1204 is configured to identify the second device in the device list acquired by the second acquisition unit 1203.
- the third acquisition unit 1205 is configured to acquire a unique identifier of the second device in the current IoT environment determined by the determination unit 1204.
- the processing unit 1206 is configured to use the unique identifier of the second device acquired by the third acquisition unit 1205 to initiate a connection request to the second device to establish a connection with the second device.
- the processing unit 1206 is further configured to, if the second acquisition unit 1203 does not acquire the device list corresponding to the type of the second device through the IoT server, notify the IoT server to establish a device list corresponding to the type of the second device, and add the feature information corresponding to the second device.
- the determination unit 1204 is specifically configured to, after the second device is determined in the device list, recognize the feature information of the second device according to the acquired image of the second device, and use the feature information of the second device to match with the feature information corresponding to each unique identifier in the device list; and if there is the feature information that matches with the feature information of the second device, determine the device identified by the unique identifier corresponding to the matched feature information as the second device.
- the processing unit 1206 is further configured to, if the determination unit 1204 determines that there is no feature information that matches with the feature information of the second device, notify the IoT server to assign a unique identifier to the device in the device list, and add the corresponding feature information.
- the first acquisition unit 1201 is further configured to, after the image of the second device is acquired through the camera, display the image on a screen; and the processing unit 1206 is further configured to, if the number of the displayed IoT devices displayed by the first acquisition unit 1201 is greater than 1, after an instruction input by a user to select any of the IoT devices is received, recognize the selected IoT object as the target device to be recognized.
- the first acquisition unit 1201 is further configured to acquire an image of a third device through the camera; and acquire an image of a fourth device through the camera;
- the recognition unit 1202 is further configured to recognize the type of the third device through the image acquired by the first acquisition unit 1201; and recognize the type of the fourth device through the acquired image;
- the second acquisition unit 1203 is further configured to acquire a device list corresponding to the type of the third device through the IoT server; and acquire a device list corresponding to the type of the fourth device through the IoT device;
- the determination unit 1204 is further configured to determine the third device in the device list corresponding to the type of the third device acquired by second acquisition unit 1203; determine the fourth device in the acquired device list corresponding to the type of the fourth device; and determine whether the fourth device serves as a device that initiates a request for establishing a connection;
- the processing unit 1206 is further configured to, if the determination unit 1204 determines that the fourth device serves as a device that initiates a request for establishing a connection
- the processing unit 1206 is further configured to, after receiving a request input by the user to acquire an IoT menu of the second device, send a request for acquiring an IoT menu of the second device to the second device; after receiving an IoT menu sent by the second device, display the IoT menu in a visual floating window; and after receiving a request by the user for selecting any menu option in the IoT menu, send the request to the second device such that the second device performs related operations according to the request.
- the first acquisition unit 1201 is configured to acquire an image of a fifth device through the camera;
- the recognition unit 1202 is configured to recognize the type of the fifth device through the image of the fifth device acquired by the first acquisition unit 1201;
- the second acquisition unit 1203 is configured to acquire, through an IoT server, a device list corresponding to the type of the fifth device recognized by the recognition unit 1202;
- the determination unit 1204 is configured to identify the fifth device in the device list acquired by the second acquisition unit 1203;
- the third acquisition unit is configured to, through the IoT server, acquire the IoT state of the fifth device determined by the determination unit 1204;
- the processing unit 1206 is further configured to display, through a visual floating window, the IoT state of the fifth device acquired by the third acquisition unit 1205.
- the processing unit 1206 is further configured to, when the IoT state of the fifth device is displayed through the visual floating window, display the IoT state in such a way as not to block the position of the image of the fifth device.
- the third acquisition unit 1205 is further configured to, if the IoT state of the fifth device determined by the determination unit 1204 is a connected state, acquire a list of the devices connected to the fifth device through the IoT server, and trigger the processing unit 1206 to display the list in the form of a floating menu at a position other than the position of the image of the fifth device.
- the above units in the embodiment may be integrated into one or may be deployed separately; or may be combined into one unit, or may be further split into multiple sub-units.
- a first device with a camera captures and recognizes other smart devices (a second device, a third device, a fourth device, a fifth device, ...) in the current IoT environment through the camera, such that the first device may quickly establish a connection with the devices (the second device, the third device, the fourth device, the fifth device, ...); or quickly establish a connection between the devices (the second device, the third device, the fourth device, the fifth device, ).
- the state of the devices (the second device, the third device, the fourth device, the fifth device ...) and an operation menu are displayed on the picture captured through the camera of the first device in augmented reality.
- the disclosure that minimizes the operation intervention by a user during the interconnection of devices is implemented in the disclosure.
- the intention of the user is intelligently recognized and automatically executed (the establishment of a connection) in the disclosure.
- a UI operation interface of augmented reality and the identification information of devices are provided in the disclosure.
- FIG. 13 is a block diagram illustrating a device 1301 in a network environment 1300 according to various embodiments.
- the device 1301 (e.g., first through fifth devices) in the network environment 1300 may communicate with a device 1302 via a first network 1398 (e.g., a short-range wireless communication network), or a device 1304 or a server 1308 via a second network 1399 (e.g., a long-range wireless communication network).
- the device 1301 may communicate with the device 1304 via the server 1308.
- the device 1301 may include a processor 1320, memory 1330, an input device 1350, a sound output device 1355, a display device 1360, an audio module 1370, a sensor module 1376, an interface 1377, a haptic module 1379, a camera module 1380, a power management module 1388, a battery 1389, a communication module 1390, a subscriber identification module (SIM) 1396, or an antenna module 1397.
- at least one (e.g., the display device 1360 or the camera module 1380) of the components may be omitted from the device 1301, or other components may be added in the device 1301.
- some of the components may be implemented as single integrated circuitry.
- the sensor module 1376 e.g., a fingerprint sensor, an iris sensor, or an illuminance sensor
- the display device 1360 e.g., a display
- an audio module 1370 e.g., a microphone
- a sensor module 1376 e.g., a fingerprint sensor
- the processor 1320 may execute, for example, software (e.g., a program 1340) to control at least one other component (e.g., a hardware or software component) of the device 1301 coupled with the processor 1320, and may perform various data processing or computation. According to one embodiment, as at least part of the data processing or computation, the processor 1320 may load a command or data received from another component (e.g., the sensor module 1376 or the communication module 1390) in volatile memory 1332, process the command or the data stored in the volatile memory 1332, and store resulting data in non-volatile memory 1334.
- software e.g., a program 1340
- the processor 1320 may load a command or data received from another component (e.g., the sensor module 1376 or the communication module 1390) in volatile memory 1332, process the command or the data stored in the volatile memory 1332, and store resulting data in non-volatile memory 1334.
- the processor 1320 may include a main processor 1321 (e.g., a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor 1323 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor 1321.
- auxiliary processor 1323 may be adapted to consume less power than the main processor 1321, or to be specific to a specified function.
- the auxiliary processor 1323 may be implemented as separate from, or as part of the main processor 1321. At least one operation or function according to various embodiments of the disclosure may be performed by the processor 1320.
- the auxiliary processor 1323 may control at least some of functions or states related to at least one (e.g., the display device 1360, the sensor module 1376, or the communication module 1390) of the components of the device 1301, instead of the main processor 1321 while the main processor 1321 is in an inactive (e.g., sleep) state or along with the main processor 1321 while the main processor 1321 is an active state (e.g., performing an application).
- the auxiliary processor 1323 e.g., an image signal processor or a communication processor
- the memory 1330 may store various data used by at least one component (e.g., the processor 1320 or the sensor module 1376) of the device 1301.
- the various data may include, for example, software (e.g., the program 1340) and input data or output data for a command related thereto.
- the memory 1330 may include the volatile memory 1332 or the non-volatile memory 1334.
- the program 1340 may be stored in the memory 1330 as software, and may include, for example, an operating system (OS) 1342, middleware 1344, or an application 1346.
- OS operating system
- middleware middleware
- application application
- the input device 1350 may receive a command or data to be used by other component (e.g., the processor 1320) of the device 1301, from the outside (e.g., a user) of the device 1301.
- the input device 1350 may include, for example, a microphone, a mouse, a keyboard, or a pen input device (e.g., a stylus pen).
- the sound output device 1355 may output sound signals to the outside of the device 1301.
- the sound output device 1355 may include, for example, a speaker or a receiver.
- the speaker may be used for general purposes, such as playing multimedia or playing record, and the receiver may be used for an incoming calls. According to an embodiment, the receiver may be implemented as separate from, or as part of the speaker.
- the display device 1360 may visually provide information to the outside (e.g., a user) of the device 1301.
- the display device 1360 may include, for example, a display, a hologram device, or a projector and control circuitry to control a corresponding one of the display, hologram device, and projector.
- the display device 1360 may include touch circuitry adapted to detect a touch, or sensor circuitry (e.g., a pressure sensor) adapted to measure the intensity of force incurred by the touch.
- the audio module 1370 may convert a sound into an electrical signal and vice versa. According to an embodiment, the audio module 1370 may obtain the sound via the input device 1350, or output the sound via the sound output device 1355 or a headphone of an external electronic device (e.g., the device 1302) directly or wirelessly coupled with the device 1301.
- an external electronic device e.g., the device 1302
- the sensor module 1376 may detect an operational state (e.g., power or temperature) of the device 1301 or an environmental state (e.g., a state of a user) external to the device 101, and then generate an electrical signal or data value corresponding to the detected state.
- the sensor module 1376 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.
- the interface 1377 may support one or more specified protocols to be used for the device 1301 to be coupled with the external electronic device (e.g., the device 1302) directly (e.g., wiredly) or wirelessly.
- the interface 1377 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.
- HDMI high definition multimedia interface
- USB universal serial bus
- SD secure digital
- a connection terminal 1378 may include a connector via which the electronic device 1301 may be physically connected with the external electronic device (e.g., the device 1302).
- the connection terminal 1378 may include, for example, a HDMI connector, a USB connector, a SD card connector, or an audio connector (e.g., a headphone connector).
- the haptic module 1379 may convert an electrical signal into a mechanical stimulus (e.g., a vibration or motion) or electrical stimulus which may be recognized by a user via his tactile sensation or kinesthetic sensation.
- the haptic module 1379 may include, for example, a motor, a piezoelectric element, or an electric stimulator.
- the camera module 1380 may capture a still image or moving images.
- the camera module 1380 may include one or more lenses, image sensors, image signal processors, or flashes.
- the power management module 1388 may manage power supplied to the device 1301. According to one embodiment, the power management module 1388 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).
- PMIC power management integrated circuit
- the battery 1389 may supply power to at least one component of the device 1301.
- the battery 1389 may include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.
- the communication module 1390 may support establishing a wired or wireless communication channel between the electronic device 1301 and an external electronic device (e.g., the device 1302, the device 1304, or the server 1308) and performing communication through the established communication channel.
- the communication module 1390 may include one or more communication processors that are operable independently from the processor 1320 (e.g., the application processor (AP)) and supports a direct (e.g., wired) communication or a wireless communication.
- AP application processor
- the communication module 1390 may include a wireless communication module 1392 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1394 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module).
- a wireless communication module 1392 e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module
- GNSS global navigation satellite system
- wired communication module 1394 e.g., a local area network (LAN) communication module or a power line communication (PLC) module.
- LAN local area network
- PLC power line communication
- a corresponding one of these communication modules may communicate with the external electronic device via the first network 1398 (e.g., a short-range communication network, such as Bluetooth TM , wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or the second network 1399 (e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., LAN or wide area network (WAN))
- the wireless communication module 1392 may identify and authenticate the device 1301 in a communication network, such as the first network 1398 or the second network 1399, using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the SIM 1396.
- subscriber information e.g., international mobile subscriber identity (IMSI)
- the antenna module 1397 may transmit or receive a signal or power to or from the outside (e.g., the external device).
- the antenna module 1397 may include one or more antennas including a radiating element composed of a conductive material or a conductive pattern formed in or on a substrate (e.g., PCB).
- the antenna module 1397 may include a plurality of antennas. In such a case, at least one antenna appropriate for a communication scheme used in the communication network, such as the first network 1398 or the second network 1399, may be selected, for example, by the communication module 1390 from the plurality of antennas. The signal or the power may then be transmitted or received between the communication module 1390 and the external electronic device via the selected at least one antenna.
- another component e.g., a radio frequency integrated circuit (RFIC)
- RFIC radio frequency integrated circuit
- At least some of the above-described components may be coupled mutually and communicate signals (e.g., commands or data) therebetween via an inter-peripheral communication scheme (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).
- an inter-peripheral communication scheme e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)
- commands or data may be transmitted or received between the device 1301 and the external device 1304 via the server 1308 coupled with the second network 1399.
- Each of the device 1302 and the device 1304 may be a device of the same type as or a different type than the device 1301.
- all or some of operations to be executed at the device 1301 may be executed at one or more of the external devices 1302, 1304, or 1308.
- the device 1301, instead of, or in addition to, executing the function or the service may request the one or more external electronic devices to perform at least part of the function or the service.
- the one or more external devices receiving the request may perform the at least part of the function or the service requested, or an additional function or an additional service related to the request, and transfer an outcome of the performing to the device 1301.
- the device 1301 may provide the outcome, with or without further processing of the outcome, as at least part of a reply to the request.
- a cloud computing, distributed computing, or client-server computing technology may be used, for example.
- the device may be one of various types of devices.
- the devices may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. According to an embodiment of the disclosure, the devices are not limited to those described above.
- each of such phrases as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C,” may include any one of, or all possible combinations of the items enumerated together in a corresponding one of the phrases.
- such terms as “1st” and “2nd,” or “first” and “second” may be used to simply distinguish a corresponding component from another, and does not limit the components in other aspect (e.g., importance or order).
- an element e.g., a first element
- the element may be coupled with the other element directly (e.g., wiredly), wirelessly, or via a third element.
- module may include a unit implemented in hardware, software, or firmware, and may interchangeably be used with other terms, for example, “logic,” “logic block,” “part,” or “circuitry”.
- a module may be a single integral component, or a minimum unit or part thereof, adapted to perform one or more functions.
- the module may be implemented in a form of an application-specific integrated circuit (ASIC).
- ASIC application-specific integrated circuit
- Various embodiments as set forth herein may be implemented as software (e.g., the program 1340) including one or more instructions that are stored in a storage medium (e.g., internal memory 1336 or external memory 1338) that is readable by a machine.
- a processor of the machine may invoke at least one of the one or more instructions stored in the storage medium, and execute it, with or without using one or more other components under the control of the processor. This allows the machine to be operated to perform at least one function according to the at least one instruction invoked.
- the one or more instructions may include a code generated by a complier or a code executable by an interpreter.
- the machine-readable storage medium may be provided in the form of a non-transitory storage medium.
- non-transitory simply means that the storage medium is a tangible device, and does not include a signal (e.g., an electromagnetic wave), but this term does not differentiate between where data is semi-permanently stored in the storage medium and where the data is temporarily stored in the storage medium.
- a signal e.g., an electromagnetic wave
- a method may be included and provided in a computer program product.
- the computer program product may be traded as a product between a seller and a buyer.
- the computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded) online via an application store (e.g., Play Store TM ), or between two user devices (e.g., smart phones) directly. If distributed online, at least part of the computer program product may be temporarily generated or at least temporarily stored in the machine-readable storage medium, such as memory of the manufacturer's server, a server of the application store, or a relay server.
- CD-ROM compact disc read only memory
- an application store e.g., Play Store TM
- two user devices e.g., smart phones
- each component e.g., a module or a program of the above-described components may include a single entity or multiple entities. According to various embodiments, one or more of the above-described components may be omitted, or one or more other components may be added. Alternatively or additionally, a plurality of components (e.g., modules or programs) may be integrated into a single component. In such a case, according to various embodiments, the integrated component may still perform one or more functions of each of the plurality of components in the same or similar manner as they are performed by a corresponding one of the plurality of components before the integration.
- operations performed by the module, the program, or another component may be carried out sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order or omitted, or one or more other operations may be added.
- the disclosure provides a method and apparatus for establishing a device connection, which can automatically establish a connection between devices with less operation on the devices.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Databases & Information Systems (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
A method and apparatus for establishing a device connection applied to a first device with a camera are provided. The method includes acquiring an image of a second device by the first device through the camera, recognizing the type of the second device through the acquired image of the second device, acquiring a device list corresponding to the type of the second device through an IoT server, and identifying the second device in the device list, acquiring a unique identifier of the second device in the current IoT environment, and using the unique identifier of the second device to initiate a connection request to the second device to establish a connection with the second device. The method may automatically establish a connection between devices with less operation of the devices.
Description
The disclosure relates to the technical field of communication. Particularly, the disclosure relates to a method and apparatus for establishing a device connection.
In the age of popularization of smart devices, with the wave of internet of things (IoT), there are more and more application scenarios of interconnection between devices. Accordingly, the requirements for intellectualization of devices are getting higher, and the requirements for the convenience of application scenarios are getting higher.
The above information is presented as background information only to assist with an understanding of the disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with regard to the disclosure.
In an IoT environment, there may be multiple connectable smart devices. How to quickly distinguish a target smart device and establish a connection is implemented in the existing implementations as follows.
A current user needs to open the Bluetooth or a wireless network first, search for a device list in the current IoT environment, select the target device in the list, try to establish a connection, and enter a connection password until the connection is successfully established.
Such a process to establish a connection obviously does not meet the requirements for the convenience of smart devices.
And for device without input/output (IO) accessories, such as a keyboard, a touch screen, a mouse, etc., or without a display screen, or with a small-size display screen, the connections with other devices cannot be established.
Aspects of the disclosure are to address at least the above-mentioned problems and/or disadvantages and to provide at least the advantages described below. Accordingly, an aspect of the disclosure is to provide an apparatus and method for establishing a device connection, which can automatically establish a connection between devices with less operation on the devices.
In order to solve the above technical problems, the technical solution of the disclosure is implemented as follows.
Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments.
In accordance with an aspect of the disclosure, a method for establishing device connections, applied to a first device with a camera is provided. The method includes acquiring an image of a second device by the first device through the camera, recognizing the type of the second device through the acquired image of the second device, acquiring a device list corresponding to the type of the second device through an Internet-of-things server, and identifying the second device in the device list, acquiring a unique identifier of the second device in the current Internet-of-things environment, and using the unique identifier of the second device to initiate a connection request to the second device to establish a connection with the second device.
In accordance with another aspect of the disclosure, a device for establishing a device connection applied to a first device is provided. The device includes a camera includes a first acquisition unit, a recognition unit, a second acquisition unit, a determination unit, a third acquisition unit, and a processing unit, in which the first acquisition unit is configured to acquire an image of the second device through the camera, the recognition unit is configured to recognize the type of the second device through the image of the second device acquired by the first acquisition unit, the second acquisition unit is configured to acquire, through an Internet-of-things server, a device list corresponding to the type of the second device recognized by the recognition unit, the determination unit is configured to identify the second device in the device list acquired by the second acquisition unit, the third acquisition unit is configured to acquire a unique identifier of the second device in the current Internet-of-things environment determined by the determination unit, and the processing unit is configured to use the unique identifier of the second device acquired by the third acquisition unit to initiate a connection request to the second device to establish a connection with the second device.
According to the above technical solution, in the disclosure, the first device captures, through the camera, and recognizes the device type of the second device to be connected, and acquires the device list corresponding to the type of the second device from the Internet-of-things server to determine the related information of the second device so as to establish a connection with the second device. Establishing a connection between devices with less operation on the devices can be implemented in this solution.
Other aspects, advantages, and salient features of the disclosure will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses various embodiments of the disclosure.
Accordingly, an aspect of the disclosure is to provide an apparatus and method for establishing a device connection, which can automatically establish a connection between devices with less operation on the devices.
The above and other aspects, features and advantages of various embodiments of the disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a flowchart of establishing a device connection according to an embodiment of the disclosure;
FIG. 2 is a diagram of a first device acquiring an image of a second device through a camera according to an embodiment of the disclosure;
FIG. 3 is a flowchart of a first device controlling a second device through an internet of things (IoT) menu according to an embodiment of the disclosure;
FIG. 4 is a diagram of initiating a request for an IoT menu by clicking according to an embodiment of the disclosure;
FIG. 5 is a diagram of displaying an IoT menu of a second device on a visual floating window on a first device according to an embodiment of the disclosure;
FIG. 6 is a diagram of selecting a mute function in an IoT menu of the second device according to an embodiment of the disclosure;
FIG. 7 is a flowchart of establishing a device connection according to an embodiment of the disclosure;
FIG. 8 is a diagram of establishing a device connection with a pair of glasses according to an embodiment of the disclosure;
FIG. 9 is a flowchart of displaying an IoT state of devices according to an embodiment of the disclosure;
FIG. 10 is a diagram of displaying an unconnected state according to an embodiment of the disclosure;
FIG. 11 is a diagram of displaying a connected state in according to an embodiment of the disclosure;
FIG. 12 is a diagram of the structure of an apparatus according to an embodiment of the disclosure; and
FIG. 13 is a block diagram of a device in a network environment according to an embodiment of the disclosure.
Throughout the drawings, like reference numerals will be understood to refer to like parts, components, and structures.
The following description with reference to the accompanying drawings is provided to assist in comprehensive understanding of various embodiments of the disclosure as defined by the claims and their equivalents. It includes various specific details to assist in that understanding but these are to be regarded as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the various embodiments described herein can be made without departing from the scope and spirit of the disclosure. In addition, descriptions of well-known functions and constructions may be omitted for clarity and conciseness.
The terms and words used in the following description and claims are not limited to the bibliographical meanings, but, are merely used by the inventor to enable a clear and consistent understanding of the disclosure. Accordingly, it should be apparent to those skilled in the art that the following description of various embodiments of the disclosure is provided for illustration purpose only and not for the purpose of limiting the disclosure as defined by the appended claims and their equivalents.
It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.
An embodiment of the disclosure provides a method for establishing a device connection which applied to a first device with a camera. In the method, the first device captures and recognizes the device type of a second device to be connected through the camera, and acquires a device list corresponding to the type of the second device from an internet of things (IoT) server to determine the related information of the second device so as to establish a connection with the second device. Automatically establishing a connection between devices in the case that the hardware requirements of the devices are not high can be implemented in this solution.
In the embodiment of the disclosure, the devices such as the first device, the second device and the like are not specifically referred to as a type of devices, but refer to any device in the current IoT environment.
In the embodiment of the disclosure, the first device is taken as a main body for controlling the connection establishment. The first device may be a mobile phone, an iPad®, glasses or the like; and the first device needs to have camera function, that is, with a camera.
The process of establishing a device connection in the embodiment of the disclosure will be described below in detail with reference to the accompanying drawings.
First Embodiment
In this embodiment, a process of one device establishing a connection with another device is taken as an example.
FIG. 1 is a flowchart of establishing a device connection in according to an embodiment of the disclosure.
Referring to FIG. 1, according to various embodiments of the disclosure, at operation 101, a first device 200 acquires an image of a second device 210 through a camera (e.g., a camera module).
According to various embodiments of the disclosure, after acquiring the image of the second device 210 through the camera, the first device 100 displays the image on a screen (e.g., a touch screen display). That is, after the first device captures the image of the second device, all the captured contents are displayed on the screen.
If the number of the displayed IoT devices is greater than 1, after an instruction input by a user to select any of the IoT devices is received, the selected IoT object is the target device to be recognized. A command for selecting any one device according to various embodiments of the disclosure may include various types of commands such as a touch (e.g., tap) gesture, a swipe gesture, a drag gesture, a hovering gesture, or a voice command.
According to various embodiments of the disclosure, when the target image is captured, multiple devices may be relatively close to each other, and the multiple devices may be captured at the same time. If the first device is moved or the user moves, there will be some inconvenience, or the second device 210 may not be accurately captured. In the disclosure, when this situation occurs, the user may manually select the target device, i.e., the second device, by touch, such that the first device 200 will know which device is to be recognized later. And if the first device 200 is a pair of glasses (e.g., Google glasses), instructions may be input through some virtual gestures or body movements.
According to various embodiments of the disclosure, at operation 102, the first device 200 recognizes a type of the second device 210 through the acquired image of the second device 210.
According to various embodiments of the disclosure, after operation 102 is performed, the first device 200 acquires a device list corresponding to the type of the second device 210 through the IoT server. After the device list is acquired, operation 103 is performed. And if the device list corresponding to the type of the second device 210 is not acquired through the IoT server, the IoT server is notified to establish a device list corresponding to the type of the second device 210, and add the feature information corresponding to the second device 210 in the established device list.
According to various embodiments of the disclosure, in the foregoing embodiment of the disclosure, notifying the second device 210 by the first device 200 to add the device list and the feature information may be implemented by connecting the first device 200 to the IoT server or may be implemented by notifying the IoT server to manually add the first device 200 at the IoT server. According to various embodiments of the disclosure, the implementation of adding a device list corresponding to one device type to the IoT server is not limited in the disclosure.
According to various embodiments of the disclosure, if the type of the second device 210 is recognized as a refrigerator and there is no device list corresponding to refrigerators, a device list of refrigerators is added to the IoT server. The function or operation of adding the device list of refrigerators may include, for example, assigning a unique identifier to the second device 210, and storing the brand of the device, the current state of the device (power-on, standby, contents being played, and the like), the name of the device, the model of the device, the surrounding environment of the device and the like. According to various embodiments of the disclosure, as for adding the feature information of the device, the feature information of the device may be added manually or by voice input, or may be added by recognizing the image of the second device 210 to acquire the feature information.
FIG. 2 is a schematic diagram of a first device acquiring an image of a second device through a camera according to an embodiment of the disclosure.
Referring to FIG. 2, the example illustrates that the first device 200 is a mobile terminal and the second device 210 is a television set. In another implementation, the first device 200 may also be a pair of glasses with a camera.
In FIG. 2, the first device 200 captures an image of the second device 210 through the camera. The device type of the second device 210 is recognized in the back-end (e.g., a server), and the second device is further identified in the corresponding device list.
Referring to FIG. 2, after the second device 210 is identified, the information about the second device is identified and displayed on the captured image in a preset mode (e.g., in a form of a dashed box on the contour boundary of the second device). According to various embodiments of the disclosure, the information may be not displayed, and the second device is directly determined in the back-end.
According to various embodiments of the disclosure, at operation 103, the first device 200 acquires a device list corresponding to a type of the second device 210 through a server (e.g., the IoT server) and identifies the second device 210 in the device list.
According to various embodiments of the disclosure, there may be related information of multiple devices or related information of one device in the device list acquired in this operation. No matter how many devices have their related information presented, the feature information of the recognized device should match and identify the second device 210.
Before this operation is performed, the feature information of the second device 210 is recognized according to the acquired image of the second device 210, which may be specifically implemented by determining the feature information of the second device 210 based on preset recognition and some local pre-configuration. The method for recognizing the feature information of the second device 210 is not limited in the disclosure. The method may be implemented according to the existing implementation. And then, the second device 210 is determined in the device list corresponding to the type of the second device 210 acquired through the IoT server, which is described as below.
According to various embodiments of the disclosure, the feature information of the second device 210 is used to match with the feature information corresponding to each unique identifier in the device list. That is, the feature information corresponding to each unique identifier is matched. And if there is the feature information that matches the feature information of the second device 210, the device identified by the unique identifier corresponding to the matched feature information is determined as the second device 210.
According to various embodiments of the disclosure, for different types of devices, matching rules may be preset to determine whether the feature information matches, and the following rules may be followed but are not limited thereto.
According to various embodiments of the disclosure, each piece of feature information of any one device is separately matched separately, and a device of which the number of matched feature information reaches the preset number serves as a device which is matched with the feature information of the second device 210, and serves as the target device. And take a television set for example, the brand of the television set, the name of the television set, the signals of the television set, the power-on state of the television set, the environment surrounding the device (such as the color of the cabinet for television set, objects around the television set, etc.) are matched separately. And if the preset value is set to 3, the device with 3 or more matched feature information will serve as the target device, that is, the second device 210 to be determined.
According to various embodiments of the disclosure, if there are multiple corresponding devices with the number of matched feature information reaches the preset number of matched feature information, the device with the most matched feature information is selected as the target device; or the device of which the feature information matches certain preset feature information is determined as the target device.
According to various embodiments of the disclosure, if there is no feature information that matches the feature information of the second device 210, that is, the devices in the device list are the same type (e.g., the same refrigerator) as the second device 210 but are not the second device 210, none of the multiple refrigerators in the refrigerator list is the refrigerator indicated by the second device 210.
According to various embodiments of the disclosure, in this case, the first device 200 notifies the IoT server to assign a unique identifier to the second device 210 in the device list, and add the corresponding feature information.
According to various embodiments of the disclosure, in the specific implementation, the first device 200 may send the feature information of the second device 210 to the IoT server, and the IoT server may add the feature information of the second device 210 to the device list corresponding to the second device 210 and assign the unique identifier.
According to various embodiments of the disclosure, the IoT server may be only notified that the new device is not in the device list, and the related information may be directly (e.g., manually) added in the device list on the IoT server side.
According to various embodiments of the disclosure, at operation 104, the first device 200 acquires a unique identifier of the second device in the current IoT environment (e.g., from the IoT server).
According to various embodiments of the disclosure, at operation 105, the first device 200 uses the unique identifier of the second device 210 to transmit (e.g., initiate) a connection request to the second device 210 to establish a connection with the second device 210.
According to various embodiments of the disclosure, during the establishment of a connection, the first device 200 and the second device 210 negotiate and select an IoT protocol supported by the two devices, and establish a connection by interaction. In the specific implementation of the disclosure, once one of the devices determines the related information of the target device, the process of negotiating and establishing a connection is similar to the existing implementation, which will not be described here in detail.
According to various embodiments of the disclosure, in this embodiment, a device with a camera may automatically establish a connection with the target device. Manually searching all the IoT devices in the current environment and manually selecting the target device and establishing a connection are not needed in this embodiment.
Second Embodiment
This embodiment describes the process of controlling one device through another device after a connection has been established between the two devices.
FIG. 3 is a flowchart of a first device controlling a second device through an IoT menu according to an embodiment of the disclosure.
Referring to FIG. 3, at operation 301, after receiving a request input by a user for acquiring an IoT menu 500 of the second device 210, the first device 200 sends, to the second device 210, a request for acquiring the IoT menu of the second device 210.
According to various embodiments of the disclosure, the method for sending a request for acquiring the IoT menu 500 may be implemented by a menu request that is triggered by preconfigured related operations. For example, if the picture of the second device is touched by clicking or sliding on the screen of the first device, the first device receives the request for the IoT menu 500 input by the user.
FIG. 4 is a diagram of initiating a request for an IoT menu by clicking according to an embodiment of the disclosure.
Referring to FIG. 4, to respond to a user's clicks to the image of the second device by a finger, at least one of the first device or the second device determines that a request is input by the user for acquiring an IoT menu 500 of the second device. Then a request for acquiring the IoT menu 500 of the second device is sent to the second device.
The IoT menu 500 herein may include a function menu of the second device. For example, the IoT menu 500 according to various embodiments of the disclosure may include at least one menu among a menu 501 for turning on/off power, menus 502 and 503 for controlling volume, menus 504 and 505 for switching channels, or a menu 506 for muting.
According to various embodiments of the disclosure, in the specific implementation, if the first device has a voice input function, the request may also be input by voice.
According to various embodiments of the disclosure, in response to a request for an IoT menu sent by the first device 200, the second device 210 sends the data related to the IoT menu to the first device 200.
According to various embodiments of the disclosure, at operation 302, after receiving an IoT menu sent by the second device 210, the first device 200 displays information about the IoT menu (e.g., a menu related to a volume control function, a menu related to channel switch function, a menu related to a device power on/off function, or a menu related to a muting function) in a visual floating window.
FIG. 5 is a diagram of displaying an IoT menu of a second device on a visual floating window on a first device according to an embodiment of the disclosure.
Referring to FIG. 5, according to various embodiments of the disclosure, when the IoT menu is displayed at a designated position, the contour boundary of the image of the second device 210 displayed on the first device 200 may be recognized, and the IoT menu of the second device is preferably displayed outside the contour boundary of the image of the second device, thus not affecting the viewing of the image of the second device.
According to various embodiments of the disclosure, at operation 303, after receiving a request by the user for selecting any menu option in the IoT menu, the first device sends the request to the second device such that the second device performs related operations according to the request.
According to various embodiments of the disclosure, any menu option may also be selected by voice or touch. If the first device is a pair of glasses, the touch control may be performed through virtual gestures and the like.
FIG. 6 is a diagram of selecting a mute menu in an IoT menu of a second device according to an embodiment of the disclosure.
Referring to FIG. 6, after receiving a request for "silence" input by a user, the first device 200 (e.g., a mobile phone) sends the request to the second device 210 (e.g., a television set), and the second device 210 mutes the television set as if it received a request for "mute" function from a remote controller.
Based on the above description, the first device 200 may control the second device 210 to perform the functional operation of the second device 210. The user may control the second device 210 in different places in a room, such as controlling a speaker to play music.
Third Embodiment
In this embodiment, a process of at least one device assisting other devices to establish a connection is taken as an example.
FIG. 7 is a flowchart of establishing a connection between devices according to an embodiment of the disclosure.
Referring to FIG. 7, according to various embodiments of the disclosure, at operation 701, the first device (e.g., the first device 200 of FIG. 1) recognizes a type of a third device 800 through a camera. The first device 200 recognizes a type of the third device 800 through the acquired image. The first device 200 acquires a device list corresponding to the type of the third device 800 through an IoT server and identifies the third device 800 in the device list.
According to various embodiments of the disclosure, at operation 702, the first device 200 captures an image of a fourth device 810 through the camera. The first device 200 recognizes a type of the fourth device 810 through the acquired image, acquires a device list corresponding to the type of the fourth device 810 through an IoT server, and identifies the fourth device 810 in the device list.
In the specific implementation of the embodiment of the disclosure, operation 701 and operation 702 are in no particular order. The specific implementation of operation 701 and operation 702 may refer to the specific implementation of operations 101 to 103.
In this embodiment, the first device 200 in this example is a pair of glasses.
FIG. 8 is a diagram of establishing a connection between devices with a pair of glasses according to an embodiment of the disclosure.
Referring to FIG. 8, the process of acquiring an image with glasses is similar to the implementation of a mobile terminal, except that the image or menu is displayed through a virtual screen. The input of instructions may be implemented through virtual gestures or voice.
Referring to FIG. 8, according to various embodiments of the disclosure, in this embodiment, the process of identifying the third device 800 and the fourth device 810 in the device list in this embodiment is similar to the process of identifying the second device 210 in the first embodiment.
According to various embodiments of the disclosure, at operation 703, the first device 200 determines whether the fourth device 810 initiates a request for establishing connection. If the fourth device 810 initiates the request for establishing connection, operation 704 is performed. Otherwise, operation 705 is performed.
According to various embodiments of the disclosure, at operation 704, the first device 200 acquires the device information of the third device 800 and sends the device information to the fourth device 810. Using the device information, the fourth device 810 initiates a connection request for establishing a connection to the third device 800 and establishes a connection with the third device 800.
According to various embodiments of the disclosure, at operation 705, the first device 200 acquires the device information of the fourth device 810 and sends the device information to the third device 800. Using the device information, the third device 800 initiates a connection request for establishing a connection to the fourth device 810 and establishes a connection with the fourth device 810.
According to various embodiments of the disclosure, the device information of the third device 800 or the fourth device 810, which are acquired by the first device 200, may be a unique identifier of the third device 800 or the fourth device 810 in the current IoT environment, such that the device that initiates the establishment of connection may initiate a request to the peer device.
According to various embodiments of the disclosure, the third device 800 or the fourth device 810 acquires the related information of the peer device. The process of connection is similar to the existing implementation. In this embodiment, the user is not required to manually select the target device in the list and is very convenient to establish a connection between devices without a display or with a display that is too small.
Fourth Embodiment
The fourth embodiment provides a process of viewing the state of the IoT device.
FIG. 9 is a flowchart of displaying an IoT state of devices according to an embodiment of the disclosure.
Referring to FIG. 9, according to various embodiments of the disclosure, at operation 901, the first device 200 captures an image of a fifth device through the camera.
According to various embodiments of the disclosure, at operation 902, the first device 200 recognizes a type of the fifth device through the acquired image.
According to various embodiments of the disclosure, at operation 903, the first device 200 acquires a device list corresponding to the type of the fifth device through the IoT server and identifies the fifth device in the device list.
According to various embodiments of the disclosure, the implementation process of operations 901 to 903 is similar to the implementation process of operations 101 to 103, which will not be described here in detail.
According to various embodiments of the disclosure, at operation 904, the first device 200 acquires an IoT state of the identified fifth device through the IoT server.
After each device is connected to another device, the other device is recorded in the IoT server. And the other device may be recorded in a device list or may be separately stored. At operation 905, the first device 200 outputs the IoT state of the fifth device through a visual floating window or auditory effects.
According to various embodiments of the disclosure, the IoT states of the device include connected and unconnected states. In this operation, when the IoT state of the fifth device is displayed through the visual floating window, the unconnected state may not be identified, or may be identified by a text, or may also be identified by a certain predetermined shape, such as a rectangular frame, a circle or the like.
According to various embodiments of the disclosure, when the unconnected state is displayed, it may be displayed without limitation, or may be displayed in such a manner as not to block the position of the image of the fifth device. For example, the unconnected state is displayed at a position other than the image of the fifth device.
FIG. 10 is a diagram of displaying an unconnected state according to an embodiment of the disclosure.
Referring to FIG. 10, for the IoT state of the fifth device, an unconnected state is identified with a dashed box displayed outside the contour boundary, but various embodiments of the disclosure are not limited thereto.
According to various embodiments of the disclosure, a connected state may be identified by text or may be identified by a certain predetermined shape, and only needs to be distinguished from the unconnected state.
According to various embodiments of the disclosure, when the connected state is displayed, it may be displayed without limited, or may be limited to be displayed at a position outside the image of the fifth device, in such a manner as not to block the fifth device.
According to various embodiments of the disclosure, for the connected state, a list of the devices connected to the fifth device is also acquired through the IoT server, and displayed in the form of a floating menu at a position other than the position of the image of the fifth device. The connected device may be displayed by name when displayed, or the unique identifier of the device may be displayed, for the purpose that the user may sure the connected devices.
FIG. 11 is a diagram of displaying a connected state according to an embodiment of the disclosure.
Referring to FIG. 11, for the IoT state of the fifth device 1100, a connected state is identified with a solid box displayed outside the contour boundary, but various embodiments of the disclosure are not limited thereto. In particular, the device list 1110 of the connected devices, specifically including Samsung® GalaxyTM S8+, Samsung® Gear2, Device2 and unnamed, is displayed outside the image of the fifth device. Herein, "connected devices" mean devices that are currently in a connected state or have ever been connected.
Referring to FIG. 11, in the embodiment of the disclosure, the first device 200 with a camera captures and recognizes other smart devices (the second device 210, the third device 800, the fourth device 810, and the fifth device 1100) in the current IoT environment through the camera, such that the first device 200 may quickly establish a connection with the devices (the second device 210, the third device 800, the fourth device 810, and the fifth device 1100); or quickly establish a connection between the devices (the second device 210, the third device 800, the fourth device 810, and the fifth device 1100). And, the state of the devices (the second device 210, the third device 800, the fourth device 810, and the fifth device 1100) and an operation menu are displayed on the picture captured through the camera of the first device in augmented reality.
According to various embodiments of the disclosure, the operation intervention by a user during the interconnection of devices is minimized in the disclosure. The intention of the user is intelligently recognized and automatically executed (the establishment of a connection), and a user interface (UI) operation interface of augmented reality and the identification information of devices are provided.
The disclosure further proposes an apparatus for establishing device connection, which is implemented by the first device 200 with a camera.
FIG. 12 is a diagram of an apparatus according to an embodiment of the disclosure.
Referring to FIG. 12, the apparatus includes a first acquisition unit 1201, a recognition unit 1202, a second acquisition unit 1203, a determination unit 1204, a third acquisition unit 1205 and a processing unit 1206. At least one operation performed by the units 1201 through 1206 according to various embodiments of the disclosure may be performed by at least one processor 1320.
Referring to FIG. 12, according to various embodiments of the disclosure, the first acquisition unit 1201 is configured to acquire an image of a second device through the camera.
According to various embodiments of the disclosure, the recognition unit 1202 is configured to recognize the type of the second device through the image of the second device acquired by the first acquisition unit 1201.
According to various embodiments of the disclosure, the second acquisition unit 1203 is configured to acquire, through an IoT server, a device list corresponding to the type of the second device recognized by the recognition unit 1202.
According to various embodiments of the disclosure, the determination unit 1204 is configured to identify the second device in the device list acquired by the second acquisition unit 1203.
According to various embodiments of the disclosure, the third acquisition unit 1205 is configured to acquire a unique identifier of the second device in the current IoT environment determined by the determination unit 1204.
According to various embodiments of the disclosure, the processing unit 1206 is configured to use the unique identifier of the second device acquired by the third acquisition unit 1205 to initiate a connection request to the second device to establish a connection with the second device.
According to various embodiments of the disclosure, the processing unit 1206 is further configured to, if the second acquisition unit 1203 does not acquire the device list corresponding to the type of the second device through the IoT server, notify the IoT server to establish a device list corresponding to the type of the second device, and add the feature information corresponding to the second device.
According to various embodiments of the disclosure, the determination unit 1204 is specifically configured to, after the second device is determined in the device list, recognize the feature information of the second device according to the acquired image of the second device, and use the feature information of the second device to match with the feature information corresponding to each unique identifier in the device list; and if there is the feature information that matches with the feature information of the second device, determine the device identified by the unique identifier corresponding to the matched feature information as the second device.
According to various embodiments of the disclosure, the processing unit 1206 is further configured to, if the determination unit 1204 determines that there is no feature information that matches with the feature information of the second device, notify the IoT server to assign a unique identifier to the device in the device list, and add the corresponding feature information.
According to various embodiments of the disclosure, the first acquisition unit 1201 is further configured to, after the image of the second device is acquired through the camera, display the image on a screen; and the processing unit 1206 is further configured to, if the number of the displayed IoT devices displayed by the first acquisition unit 1201 is greater than 1, after an instruction input by a user to select any of the IoT devices is received, recognize the selected IoT object as the target device to be recognized.
According to various embodiments of the disclosure, the first acquisition unit 1201 is further configured to acquire an image of a third device through the camera; and acquire an image of a fourth device through the camera; the recognition unit 1202 is further configured to recognize the type of the third device through the image acquired by the first acquisition unit 1201; and recognize the type of the fourth device through the acquired image; and the second acquisition unit 1203 is further configured to acquire a device list corresponding to the type of the third device through the IoT server; and acquire a device list corresponding to the type of the fourth device through the IoT device; the determination unit 1204 is further configured to determine the third device in the device list corresponding to the type of the third device acquired by second acquisition unit 1203; determine the fourth device in the acquired device list corresponding to the type of the fourth device; and determine whether the fourth device serves as a device that initiates a request for establishing a connection; and the processing unit 1206 is further configured to, if the determination unit 1204 determines that the fourth device serves as a device that initiates a request for establishing a connection, acquire the device information of the third device and send the device information to the fourth device, such that the fourth device initiates a request for establishing a connection to the third device and establishes a connection with the third device.
According to various embodiments of the disclosure, the processing unit 1206 is further configured to, after receiving a request input by the user to acquire an IoT menu of the second device, send a request for acquiring an IoT menu of the second device to the second device; after receiving an IoT menu sent by the second device, display the IoT menu in a visual floating window; and after receiving a request by the user for selecting any menu option in the IoT menu, send the request to the second device such that the second device performs related operations according to the request.
According to various embodiments of the disclosure, the first acquisition unit 1201 is configured to acquire an image of a fifth device through the camera; the recognition unit 1202 is configured to recognize the type of the fifth device through the image of the fifth device acquired by the first acquisition unit 1201; the second acquisition unit 1203 is configured to acquire, through an IoT server, a device list corresponding to the type of the fifth device recognized by the recognition unit 1202; the determination unit 1204 is configured to identify the fifth device in the device list acquired by the second acquisition unit 1203; the third acquisition unit is configured to, through the IoT server, acquire the IoT state of the fifth device determined by the determination unit 1204; and the processing unit 1206 is further configured to display, through a visual floating window, the IoT state of the fifth device acquired by the third acquisition unit 1205.
According to various embodiments of the disclosure, the processing unit 1206 is further configured to, when the IoT state of the fifth device is displayed through the visual floating window, display the IoT state in such a way as not to block the position of the image of the fifth device.
According to various embodiments of the disclosure, the third acquisition unit 1205 is further configured to, if the IoT state of the fifth device determined by the determination unit 1204 is a connected state, acquire a list of the devices connected to the fifth device through the IoT server, and trigger the processing unit 1206 to display the list in the form of a floating menu at a position other than the position of the image of the fifth device.
According to various embodiments of the disclosure, the above units in the embodiment may be integrated into one or may be deployed separately; or may be combined into one unit, or may be further split into multiple sub-units.
According to various embodiments of the disclosure, in the disclosure, a first device with a camera captures and recognizes other smart devices (a second device, a third device, a fourth device, a fifth device, ...) in the current IoT environment through the camera, such that the first device may quickly establish a connection with the devices (the second device, the third device, the fourth device, the fifth device, ...); or quickly establish a connection between the devices (the second device, the third device, the fourth device, the fifth device, ...). Also, the state of the devices (the second device, the third device, the fourth device, the fifth device ...) and an operation menu are displayed on the picture captured through the camera of the first device in augmented reality.
The disclosure that minimizes the operation intervention by a user during the interconnection of devices is implemented in the disclosure. The intention of the user is intelligently recognized and automatically executed (the establishment of a connection) in the disclosure. A UI operation interface of augmented reality and the identification information of devices are provided in the disclosure.
FIG. 13 is a block diagram illustrating a device 1301 in a network environment 1300 according to various embodiments.
Referring to FIG. 13, the device 1301 (e.g., first through fifth devices) in the network environment 1300 may communicate with a device 1302 via a first network 1398 (e.g., a short-range wireless communication network), or a device 1304 or a server 1308 via a second network 1399 (e.g., a long-range wireless communication network). According to an embodiment, the device 1301 may communicate with the device 1304 via the server 1308. According to an embodiment, the device 1301 may include a processor 1320, memory 1330, an input device 1350, a sound output device 1355, a display device 1360, an audio module 1370, a sensor module 1376, an interface 1377, a haptic module 1379, a camera module 1380, a power management module 1388, a battery 1389, a communication module 1390, a subscriber identification module (SIM) 1396, or an antenna module 1397. In some embodiments, at least one (e.g., the display device 1360 or the camera module 1380) of the components may be omitted from the device 1301, or other components may be added in the device 1301. In some embodiments, some of the components may be implemented as single integrated circuitry. For example, the sensor module 1376 (e.g., a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented as embedded in the display device 1360 (e.g., a display).
The processor 1320 may execute, for example, software (e.g., a program 1340) to control at least one other component (e.g., a hardware or software component) of the device 1301 coupled with the processor 1320, and may perform various data processing or computation. According to one embodiment, as at least part of the data processing or computation, the processor 1320 may load a command or data received from another component (e.g., the sensor module 1376 or the communication module 1390) in volatile memory 1332, process the command or the data stored in the volatile memory 1332, and store resulting data in non-volatile memory 1334. According to an embodiment, the processor 1320 may include a main processor 1321 (e.g., a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor 1323 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor 1321. Additionally or alternatively, the auxiliary processor 1323 may be adapted to consume less power than the main processor 1321, or to be specific to a specified function. The auxiliary processor 1323 may be implemented as separate from, or as part of the main processor 1321. At least one operation or function according to various embodiments of the disclosure may be performed by the processor 1320.
The auxiliary processor 1323 may control at least some of functions or states related to at least one (e.g., the display device 1360, the sensor module 1376, or the communication module 1390) of the components of the device 1301, instead of the main processor 1321 while the main processor 1321 is in an inactive (e.g., sleep) state or along with the main processor 1321 while the main processor 1321 is an active state (e.g., performing an application). According to an embodiment, the auxiliary processor 1323 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module 1380 or the communication module 1390) functionally related to the auxiliary processor 1323.
The memory 1330 may store various data used by at least one component (e.g., the processor 1320 or the sensor module 1376) of the device 1301. The various data may include, for example, software (e.g., the program 1340) and input data or output data for a command related thereto. The memory 1330 may include the volatile memory 1332 or the non-volatile memory 1334.
The program 1340 may be stored in the memory 1330 as software, and may include, for example, an operating system (OS) 1342, middleware 1344, or an application 1346.
The input device 1350 may receive a command or data to be used by other component (e.g., the processor 1320) of the device 1301, from the outside (e.g., a user) of the device 1301. The input device 1350 may include, for example, a microphone, a mouse, a keyboard, or a pen input device (e.g., a stylus pen).
The sound output device 1355 may output sound signals to the outside of the device 1301. The sound output device 1355 may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or playing record, and the receiver may be used for an incoming calls. According to an embodiment, the receiver may be implemented as separate from, or as part of the speaker.
The display device 1360 may visually provide information to the outside (e.g., a user) of the device 1301. The display device 1360 may include, for example, a display, a hologram device, or a projector and control circuitry to control a corresponding one of the display, hologram device, and projector. According to an embodiment, the display device 1360 may include touch circuitry adapted to detect a touch, or sensor circuitry (e.g., a pressure sensor) adapted to measure the intensity of force incurred by the touch.
The audio module 1370 may convert a sound into an electrical signal and vice versa. According to an embodiment, the audio module 1370 may obtain the sound via the input device 1350, or output the sound via the sound output device 1355 or a headphone of an external electronic device (e.g., the device 1302) directly or wirelessly coupled with the device 1301.
The sensor module 1376 may detect an operational state (e.g., power or temperature) of the device 1301 or an environmental state (e.g., a state of a user) external to the device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 1376 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.
The interface 1377 may support one or more specified protocols to be used for the device 1301 to be coupled with the external electronic device (e.g., the device 1302) directly (e.g., wiredly) or wirelessly. According to an embodiment, the interface 1377 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.
A connection terminal 1378 may include a connector via which the electronic device 1301 may be physically connected with the external electronic device (e.g., the device 1302). According to an embodiment, the connection terminal 1378 may include, for example, a HDMI connector, a USB connector, a SD card connector, or an audio connector (e.g., a headphone connector).
The haptic module 1379 may convert an electrical signal into a mechanical stimulus (e.g., a vibration or motion) or electrical stimulus which may be recognized by a user via his tactile sensation or kinesthetic sensation. According to an embodiment, the haptic module 1379 may include, for example, a motor, a piezoelectric element, or an electric stimulator.
The camera module 1380 may capture a still image or moving images. According to an embodiment, the camera module 1380 may include one or more lenses, image sensors, image signal processors, or flashes.
The power management module 1388 may manage power supplied to the device 1301. According to one embodiment, the power management module 1388 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).
The battery 1389 may supply power to at least one component of the device 1301. According to an embodiment, the battery 1389 may include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.
The communication module 1390 may support establishing a wired or wireless communication channel between the electronic device 1301 and an external electronic device (e.g., the device 1302, the device 1304, or the server 1308) and performing communication through the established communication channel. The communication module 1390 may include one or more communication processors that are operable independently from the processor 1320 (e.g., the application processor (AP)) and supports a direct (e.g., wired) communication or a wireless communication. According to an embodiment, the communication module 1390 may include a wireless communication module 1392 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1394 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate with the external electronic device via the first network 1398 (e.g., a short-range communication network, such as BluetoothTM, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or the second network 1399 (e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)) These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multi components (e.g., multi chips) separate from each other. The wireless communication module 1392 may identify and authenticate the device 1301 in a communication network, such as the first network 1398 or the second network 1399, using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the SIM 1396.
The antenna module 1397 may transmit or receive a signal or power to or from the outside (e.g., the external device). According to an embodiment, the antenna module 1397 may include one or more antennas including a radiating element composed of a conductive material or a conductive pattern formed in or on a substrate (e.g., PCB). According to an embodiment, the antenna module 1397 may include a plurality of antennas. In such a case, at least one antenna appropriate for a communication scheme used in the communication network, such as the first network 1398 or the second network 1399, may be selected, for example, by the communication module 1390 from the plurality of antennas. The signal or the power may then be transmitted or received between the communication module 1390 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as part of the antenna module 1397.
At least some of the above-described components may be coupled mutually and communicate signals (e.g., commands or data) therebetween via an inter-peripheral communication scheme (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).
According to various embodiments of the disclosure, commands or data may be transmitted or received between the device 1301 and the external device 1304 via the server 1308 coupled with the second network 1399. Each of the device 1302 and the device 1304 may be a device of the same type as or a different type than the device 1301. According to an embodiment, all or some of operations to be executed at the device 1301 may be executed at one or more of the external devices 1302, 1304, or 1308. For example, if the device 1301 should perform a function or a service automatically, or in response to a request from a user or another device, the device 1301, instead of, or in addition to, executing the function or the service, may request the one or more external electronic devices to perform at least part of the function or the service. The one or more external devices receiving the request may perform the at least part of the function or the service requested, or an additional function or an additional service related to the request, and transfer an outcome of the performing to the device 1301. The device 1301 may provide the outcome, with or without further processing of the outcome, as at least part of a reply to the request. To that end, a cloud computing, distributed computing, or client-server computing technology may be used, for example.
The device according to various embodiments of the disclosure may be one of various types of devices. The devices may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. According to an embodiment of the disclosure, the devices are not limited to those described above.
It should be appreciated that various embodiments of the disclosure and the terms used therein are not intended to limit the technological features set forth herein to particular embodiments and include various changes, equivalents, or replacements for a corresponding embodiment. With regard to the description of the drawings, similar reference numerals may be used to refer to similar or related elements. It is to be understood that a singular form of a noun corresponding to an item may include one or more of the things, unless the relevant context clearly indicates otherwise. As used herein, each of such phrases as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C," may include any one of, or all possible combinations of the items enumerated together in a corresponding one of the phrases. As used herein, such terms as "1st" and "2nd," or "first" and "second" may be used to simply distinguish a corresponding component from another, and does not limit the components in other aspect (e.g., importance or order). It is to be understood that if an element (e.g., a first element) is referred to, with or without the term "operatively" or "communicatively", as "coupled with," "coupled to," "connected with," or "connected to" another element (e.g., a second element), it means that the element may be coupled with the other element directly (e.g., wiredly), wirelessly, or via a third element.
As used herein, the term "module" or "unit" may include a unit implemented in hardware, software, or firmware, and may interchangeably be used with other terms, for example, "logic," "logic block," "part," or "circuitry". A module may be a single integral component, or a minimum unit or part thereof, adapted to perform one or more functions. For example, according to an embodiment, the module may be implemented in a form of an application-specific integrated circuit (ASIC).
Various embodiments as set forth herein may be implemented as software (e.g., the program 1340) including one or more instructions that are stored in a storage medium (e.g., internal memory 1336 or external memory 1338) that is readable by a machine. For example, a processor of the machine may invoke at least one of the one or more instructions stored in the storage medium, and execute it, with or without using one or more other components under the control of the processor. This allows the machine to be operated to perform at least one function according to the at least one instruction invoked. The one or more instructions may include a code generated by a complier or a code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Wherein, the term "non-transitory" simply means that the storage medium is a tangible device, and does not include a signal (e.g., an electromagnetic wave), but this term does not differentiate between where data is semi-permanently stored in the storage medium and where the data is temporarily stored in the storage medium.
According to an embodiment, a method according to various embodiments of the disclosure may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded) online via an application store (e.g., Play StoreTM), or between two user devices (e.g., smart phones) directly. If distributed online, at least part of the computer program product may be temporarily generated or at least temporarily stored in the machine-readable storage medium, such as memory of the manufacturer's server, a server of the application store, or a relay server.
According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single entity or multiple entities. According to various embodiments, one or more of the above-described components may be omitted, or one or more other components may be added. Alternatively or additionally, a plurality of components (e.g., modules or programs) may be integrated into a single component. In such a case, according to various embodiments, the integrated component may still perform one or more functions of each of the plurality of components in the same or similar manner as they are performed by a corresponding one of the plurality of components before the integration. According to various embodiments, operations performed by the module, the program, or another component may be carried out sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order or omitted, or one or more other operations may be added.
As is apparent from the foregoing description, according to various embodiments of the disclosure, the disclosure provides a method and apparatus for establishing a device connection, which can automatically establish a connection between devices with less operation on the devices.
Effects according to various embodiments are not limited to the described effects, and it would be obvious to those of ordinary skill in the art that various effects are included in the disclosure.
While the disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the disclosure as defined by the appended claims and their equivalents.
Claims (15)
- A method for establishing a device connection by a first device, the method comprising:acquiring an image of a second device by the first device using a camera of the first device;recognizing the second device in the image; andtransmitting a request to establish a connection with the second device, the request including a unique identifier of the second device.
- The method of claim 1,wherein the recognizing of the second device comprises:requesting a device list based on a type of the second device from an internet of things (IoT) server,when the device list is stored by the IoT server, receiving the device list and identifying the second device in the device list, andwhen the device list is not stored by the IoT server, transmitting a request to the IoT server, andwherein, in response to the request, the IoT server establishes a device list corresponding to the type of the second device and adds feature information corresponding to the second device.
- The method of claim 2,wherein the recognizing of the second device comprises recognizing the feature information of the second device, andwherein the identifying of the second device in the device list further comprises:using the feature information of the second device to match an item in the device list, andwhen a first item that matches the feature information of the second device, determining the unique identifier corresponding to the second device from the first item.
- The method of claim 3, further comprising:when no item in the device list matches the feature information of the second device, transmitting a request to the IoT server,wherein, in response to the request, the IoT server assigns the unique identifier to the second device in the device list and adds an item to the device list including the feature information of the second device and the unique identifier of the second device.
- The method of claim 1, further comprising:displaying the image on a screen of the first device;determining a number of internet of things (IoT) devices in the image; andwhen the number of the IoT devices is greater than 1, receiving an input selecting a first IoT device in the image.
- The method of claim 1, further comprising:acquiring a second image of a third device by the first device using the camera;recognizing a type of the third device in the second image;receiving a first device list based on the type of the third device from an internet of things (IoT) server;determining a first item in the first device list corresponding to the third device;acquiring a third image of a fourth device using the camera;recognizing a type of the fourth device in the third image;receiving a second device list based on a type of the fourth device from the IoT server;determining a second item in the second device list corresponding to the fourth device; andwhen the fourth device is determined to initiate a request for establishing a connection, transmitting device information of the third device to the fourth device for establishing a connection with the third device.
- The method of claim 1, further comprising:receiving an input for acquiring a menu of commands for the second device;in response to the input, transmitting a request to the second device for the menu of commands;receiving and displaying the menu of commands in a floating window; andwhen an input of a command on the menu is received, transmitting a request to the second device to perform the command.
- The method of claim 1, further comprising:acquiring a second image of a fifth device by the first device using the camera;displaying the second image, wherein the fifth device is displayed in a first region;recognizing a type of the fifth device in the second image;transmitting a request for a device list based on the type of the fifth device from an internet of things (IoT) server;receiving and identifying the fifth device in the device list;receiving information including an IoT state of the fifth device from the IoT server; anddisplaying the IoT state of the fifth device through a floating window, wherein the floating window is displayed in a second region that is different from the first region,wherein the method is further comprising when the IoT state corresponds to a connected state, requesting a list of devices connected to the fifth device from the IoT server, and displaying a floating menu including the list of devices in a second region that is different from the first region.
- A device comprising:at least one camera;a touch screen display; andat least one processor,wherein the at least one processor is configured to:acquire an image of a second device through the at least one camera,recognize the second device in the image, andtransmit a request to establish a connection with the second device, the request including a unique identifier of the second device.
- The device of claim 9,wherein the at least one processor is further configured to:request a device list based on a type of the second device from an internet of things (IoT) server,when the device list is stored by the IoT server, receive the device list and identify the second device in the device list, andwhen the device list is not stored by the IoT server, transmit a request to the IoT server,wherein, in response to the request, the IoT server establishes a device list corresponding to the type of the second device and adds feature information corresponding to the second device.
- The device of claim 10,wherein, when the at least one processor recognizes the second device in the image, the at least one processor is further configured to recognized the feature information, andwherein, when the at least one processor identifies the second device in the device list, the at least one processor is further configured to:use the feature information of the second device to match at item in the device list, andwhen a first item matches the feature information of the second device, determine the unique identifier corresponding to the second device from the first item,wherein the at least one processor is further configured to, when no item in the device list matches the feature information of the second device, transmit a request to the IoT server, and wherein, in response to the request, the IoT server assigns the unique identifier to the second device in the device list and adds an item to the device list including the feature information of the second device and the unique identifier of the second device.
- The device of claim 9, wherein the at least one processor is further configured to:display the image on the touch screen display,determine a number of internet of things (IoT) devices in the image, andwhen the number of the IoT devices is greater than 1, receive an input selecting a first IoT device in the image.
- The device of claim 9, wherein the at least one processor is further configured to:acquire a second image of a third device using the at least one camera,recognize a type of the third device in the second image,receive a first device list based on the type of the third device through an internet of things (IoT) server,determine a first item in the first device list corresponding to the third device,acquire a third image of a fourth device using the at least one camera,recognize a type of the fourth device in the third image,receive a second device list based on a type of the fourth device from the IoT server,determine a second item in the second device list corresponding to the fourth device, andwhen the fourth device is determined to initiate a request for establishing a connection, transmit device information of the third device for establishing a connection with the third device.
- The device of claim 9, wherein the at least one processor is further configured to:receiving an input for acquiring an a menu of commands for the second device,in response to the input, transmit a request to the second device for the menu of commands,receiving and displaying the menu of commands in a floating window, andwhen an input of a command on the menu is received, transmit a request to the second device to perform the command.
- The device of claim 9, wherein the at least one processor is further configured to:acquire a second image of a fifth device using the at least one camera,displaying the second image, wherein the fifth device is displayed in a first region,recognize a type of the fifth device in the second image,transmit a request for a device list based on the type of the fifth device from an internet of things (IoT) server,receive and identify the fifth device in the device list,receiving information including an IoT state of the fifth device from the IoT server, anddisplay the IoT state of the fifth device through a floating window, wherein the floating window is displayed in a second region that is different from the first region,wherein the at least one processor is further configured to when the IoT state corresponds to a connected state, request a list of devices connected to the fifth device from the IoT server, and display a floating menu including the list of devices in a second region that is different from the first region.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP19840103.6A EP3808097A4 (en) | 2018-07-25 | 2019-07-24 | Method and apparatus for establishing device connection |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810824343.8A CN109068161B (en) | 2018-07-25 | 2018-07-25 | Equipment connection establishing method and device |
CN201810824343.8 | 2018-07-25 | ||
KR10-2019-0077307 | 2019-06-27 | ||
KR1020190077307A KR102630452B1 (en) | 2018-07-25 | 2019-06-27 | Method and Apparatus for Establishing Device Connection |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020022780A1 true WO2020022780A1 (en) | 2020-01-30 |
Family
ID=69178337
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/KR2019/009198 WO2020022780A1 (en) | 2018-07-25 | 2019-07-24 | Method and apparatus for establishing device connection |
Country Status (2)
Country | Link |
---|---|
US (1) | US11350264B2 (en) |
WO (1) | WO2020022780A1 (en) |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11351472B2 (en) | 2016-01-19 | 2022-06-07 | Disney Enterprises, Inc. | Systems and methods for using a gyroscope to change the resistance of moving a virtual weapon |
US11663783B2 (en) | 2016-02-10 | 2023-05-30 | Disney Enterprises, Inc. | Systems and methods for using augmented reality with the internet of things |
US10725629B2 (en) * | 2018-06-25 | 2020-07-28 | Google Llc | Identifying and controlling smart devices |
US10974132B2 (en) | 2018-10-02 | 2021-04-13 | Disney Enterprises, Inc. | Systems and methods to provide a shared interactive experience across multiple presentation devices based on detection of one or more extraterrestrial bodies |
US11014008B2 (en) | 2019-03-27 | 2021-05-25 | Disney Enterprises, Inc. | Systems and methods for game profile development based on virtual and/or real activities |
US10916061B2 (en) * | 2019-04-24 | 2021-02-09 | Disney Enterprises, Inc. | Systems and methods to synchronize real-world motion of physical objects with presentation of virtual content |
CN112804730B (en) * | 2021-01-25 | 2023-09-08 | Oppo广东移动通信有限公司 | Equipment interconnection method, device, server, intelligent equipment and storage medium |
JP2023131267A (en) * | 2022-03-09 | 2023-09-22 | 富士フイルム株式会社 | Content provision apparatus, content provision method, program, and recording medium |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130223279A1 (en) | 2012-02-24 | 2013-08-29 | Peerapol Tinnakornsrisuphap | Sensor based configuration and control of network devices |
WO2013154476A1 (en) | 2012-04-12 | 2013-10-17 | Telefonaktiebolaget L M Ericsson (Publ) | Pairing a mobile terminal with a wireless device |
US20150130957A1 (en) * | 2013-11-14 | 2015-05-14 | Qualcomm Incorporated | Method and apparatus for identifying a physical iot device |
KR20160066292A (en) * | 2014-12-02 | 2016-06-10 | 한국전자통신연구원 | System and method for controlling Internet-of-Things device |
KR20170016744A (en) * | 2015-08-04 | 2017-02-14 | 서울과학기술대학교 산학협력단 | Apparatus and method for remote controlling device using metadata |
KR20170040492A (en) * | 2015-10-05 | 2017-04-13 | 한국전자통신연구원 | Method and apparatus for registering device for smart service environment |
EP3168730A2 (en) | 2015-11-11 | 2017-05-17 | LG Electronics Inc. | Mobile terminal and method for controlling the same |
US20170195424A1 (en) * | 2015-12-30 | 2017-07-06 | Verizon Patent And Licensing Inc. | INTERNET OF THINGS (IoT) DEVICE ACTIVATION AND MANAGEMENT |
Family Cites Families (39)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR101377885B1 (en) * | 2007-07-23 | 2014-03-24 | 엘지전자 주식회사 | Appartus and method for labal display in display device |
US8090616B2 (en) * | 2008-09-08 | 2012-01-03 | Proctor Jr James Arthur | Visual identification information used as confirmation in a wireless communication |
CN102301405B (en) | 2009-01-29 | 2014-01-15 | 丰田自动车株式会社 | Object Recognition Device And Object Recognition Method |
US8818274B2 (en) * | 2009-07-17 | 2014-08-26 | Qualcomm Incorporated | Automatic interfacing between a master device and object device |
CN101794381A (en) | 2010-02-26 | 2010-08-04 | 智慧天下(北京)科技有限公司 | Method for image acquisition and discrimination by utilizing camera |
CN102377621B (en) | 2010-08-16 | 2014-12-10 | Tcl集团股份有限公司 | Equipment interconnection system and method |
JP5739722B2 (en) | 2011-04-26 | 2015-06-24 | 株式会社日立情報通信エンジニアリング | Object recognition method and recognition apparatus |
CN102760227B (en) | 2012-03-06 | 2015-04-29 | 联想(北京)有限公司 | Electronic equipment and recognition method thereof |
JP6380408B2 (en) * | 2013-01-09 | 2018-08-29 | ソニー株式会社 | Mobile device and method for establishing a wireless link |
CN106793377B (en) * | 2013-02-20 | 2019-03-26 | 松下电器(美国)知识产权公司 | The control method and recording medium of information terminal |
CN103108167A (en) | 2013-02-26 | 2013-05-15 | 深圳市高斯贝尔家居智能电子有限公司 | Method for camera connecting to local wireless network and monitoring method of wireless network |
US9367144B2 (en) * | 2013-03-13 | 2016-06-14 | Google Inc. | Methods, systems, and media for providing a remote control interface for a media playback device |
CN104063040A (en) | 2013-03-18 | 2014-09-24 | 维沃移动通信有限公司 | Intelligent control method and system for object detection and identification by front-facing camera in movable handheld equipment |
US9787498B2 (en) * | 2013-04-16 | 2017-10-10 | Throughtek Co., Ltd. | System and method of identifying networked device for establishing a P2P connection |
US9438440B2 (en) * | 2013-07-29 | 2016-09-06 | Qualcomm Incorporated | Proximity detection of internet of things (IoT) devices using sound chirps |
US9400943B2 (en) * | 2013-08-02 | 2016-07-26 | Qualcomm Incorporated | Identifying IoT devices/objects/people using out-of-band signaling/metadata in conjunction with optical images |
US20160217617A1 (en) * | 2013-08-30 | 2016-07-28 | Hewlett-Packard Development Company, L.P. | Augmented reality device interfacing |
KR101460570B1 (en) * | 2013-10-14 | 2014-11-13 | 현대모비스(주) | Camera Position Recognition System |
KR102238775B1 (en) * | 2014-02-10 | 2021-04-09 | 삼성전자주식회사 | Apparatus and method for devices administration using augmented reality in electronics device |
US20150358777A1 (en) * | 2014-06-04 | 2015-12-10 | Qualcomm Incorporated | Generating a location profile of an internet of things device based on augmented location information associated with one or more nearby internet of things devices |
US9332578B2 (en) * | 2014-07-30 | 2016-05-03 | Google Technology Holdings LLC | Connecting wireless devices using visual image capture and processing |
CN104598589A (en) | 2015-01-20 | 2015-05-06 | 惠州Tcl移动通信有限公司 | Intelligent tourist method and system based on image identification |
CN104616023A (en) | 2015-02-06 | 2015-05-13 | 安徽大学 | Object outline detection and recognition system and object outline recognition method |
WO2016157076A1 (en) * | 2015-03-30 | 2016-10-06 | Ghoson Ziad | Information processing system and method using image recognition |
CN105306450B (en) | 2015-09-28 | 2020-08-11 | 深圳新创客电子科技有限公司 | Equipment interconnection method, terminal, server and system |
US10715599B2 (en) * | 2015-11-30 | 2020-07-14 | Verizon Patent And Licensing, Inc. | Internet of things (IoT) platform and application framework |
US9860213B2 (en) * | 2015-12-30 | 2018-01-02 | Iomaxis, Llc | Method and system for securing and protecting smart devices within the internet of things ecosystem |
WO2017165705A1 (en) * | 2016-03-23 | 2017-09-28 | Bent Image Lab, Llc | Augmented reality for the internet of things |
JP6090511B1 (en) * | 2016-05-06 | 2017-03-08 | 富士ゼロックス株式会社 | Terminal device and program |
US10110678B2 (en) * | 2016-08-19 | 2018-10-23 | Sony Corporation | System and method for data communication based on image processing |
CN107038408A (en) * | 2017-01-11 | 2017-08-11 | 阿里巴巴集团控股有限公司 | Image-recognizing method and device based on augmented reality |
US10602046B2 (en) * | 2017-07-11 | 2020-03-24 | Htc Corporation | Mobile device and control method |
US11157442B2 (en) * | 2017-07-14 | 2021-10-26 | Ricoh Company, Ltd. | Information processing apparatus, information processing system, and information processing method |
JP6493466B2 (en) * | 2017-07-28 | 2019-04-03 | 富士ゼロックス株式会社 | Information processing apparatus and program |
US10620721B2 (en) * | 2018-01-29 | 2020-04-14 | Google Llc | Position-based location indication and device control |
US10650597B2 (en) * | 2018-02-06 | 2020-05-12 | Servicenow, Inc. | Augmented reality assistant |
JP7172071B2 (en) * | 2018-03-15 | 2022-11-16 | 富士フイルムビジネスイノベーション株式会社 | Information gathering device and program |
US10565764B2 (en) * | 2018-04-09 | 2020-02-18 | At&T Intellectual Property I, L.P. | Collaborative augmented reality system |
US10725629B2 (en) * | 2018-06-25 | 2020-07-28 | Google Llc | Identifying and controlling smart devices |
-
2019
- 2019-07-24 WO PCT/KR2019/009198 patent/WO2020022780A1/en unknown
- 2019-07-24 US US16/520,557 patent/US11350264B2/en active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130223279A1 (en) | 2012-02-24 | 2013-08-29 | Peerapol Tinnakornsrisuphap | Sensor based configuration and control of network devices |
WO2013154476A1 (en) | 2012-04-12 | 2013-10-17 | Telefonaktiebolaget L M Ericsson (Publ) | Pairing a mobile terminal with a wireless device |
US20150130957A1 (en) * | 2013-11-14 | 2015-05-14 | Qualcomm Incorporated | Method and apparatus for identifying a physical iot device |
KR20160066292A (en) * | 2014-12-02 | 2016-06-10 | 한국전자통신연구원 | System and method for controlling Internet-of-Things device |
KR20170016744A (en) * | 2015-08-04 | 2017-02-14 | 서울과학기술대학교 산학협력단 | Apparatus and method for remote controlling device using metadata |
KR20170040492A (en) * | 2015-10-05 | 2017-04-13 | 한국전자통신연구원 | Method and apparatus for registering device for smart service environment |
EP3168730A2 (en) | 2015-11-11 | 2017-05-17 | LG Electronics Inc. | Mobile terminal and method for controlling the same |
US20170195424A1 (en) * | 2015-12-30 | 2017-07-06 | Verizon Patent And Licensing Inc. | INTERNET OF THINGS (IoT) DEVICE ACTIVATION AND MANAGEMENT |
Non-Patent Citations (1)
Title |
---|
See also references of EP3808097A4 |
Also Published As
Publication number | Publication date |
---|---|
US11350264B2 (en) | 2022-05-31 |
US20200037144A1 (en) | 2020-01-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020022780A1 (en) | Method and apparatus for establishing device connection | |
EP3808097A1 (en) | Method and apparatus for establishing device connection | |
WO2020085789A1 (en) | Foldable electronic device for controlling user interface and operating method thereof | |
WO2020032510A1 (en) | Electronic device including electronic pen and method of controlling communication connection between electronic device and electronic pen | |
WO2020162709A1 (en) | Electronic device for providing graphic data based on voice and operating method thereof | |
WO2021096282A1 (en) | Electronic device for managing coexistence of multiple communication schemes and operating method thereof | |
WO2020067639A1 (en) | Electronic device for pairing with stylus pen and method thereof | |
WO2020166911A1 (en) | Screen providing method and electronic device supporting same | |
WO2020032443A1 (en) | Electronic device supporting personalized device connection and method thereof | |
WO2019112373A1 (en) | Method for providing artificial intelligence service during phone call and electronic device thereof | |
WO2020166894A1 (en) | Electronic device and method for recommending word in same | |
WO2020032465A1 (en) | Method for contents playback with continuity and electronic device therefor | |
WO2020106019A1 (en) | Electronic device and method for providing in-vehicle infotainment service | |
WO2019039729A1 (en) | Method for changing the size of the content displayed on display and electronic device thereof | |
WO2019190024A1 (en) | Mobile electronic device and method for forwarding user input to application according to input means | |
WO2020166932A1 (en) | Method for sharing bluetooth communication information and electronic device therefor | |
WO2020171608A1 (en) | Electronic device for providing handwriting input function and method of operating the same | |
WO2020153818A1 (en) | Method of controlling external electronic device and electronic device for supporting same | |
WO2020171342A1 (en) | Electronic device for providing visualized artificial intelligence service on basis of information about external object, and operating method for electronic device | |
WO2020032541A1 (en) | Method for processing incoming call and electronic device for supporting the same | |
WO2020180008A1 (en) | Method for processing plans having multiple end points and electronic device applying the same method | |
WO2020022774A1 (en) | Method of controlling operation mode using electronic pen and electronic device for same | |
WO2020032650A1 (en) | Speech recognition method and electronic device for supporting the same | |
WO2020032512A1 (en) | Electronic device and method for displaying affordance for providing charging of battery of external device through display | |
WO2019156411A1 (en) | Electronic device for connecting with external devices based on connection information and operating method thereof |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19840103 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2019840103 Country of ref document: EP Effective date: 20210118 |