WO2021024753A1 - Information processing device, information processing method, and program - Google Patents

Information processing device, information processing method, and program Download PDF

Info

Publication number
WO2021024753A1
WO2021024753A1 PCT/JP2020/027777 JP2020027777W WO2021024753A1 WO 2021024753 A1 WO2021024753 A1 WO 2021024753A1 JP 2020027777 W JP2020027777 W JP 2020027777W WO 2021024753 A1 WO2021024753 A1 WO 2021024753A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
vibration
information processing
vibration data
difference
Prior art date
Application number
PCT/JP2020/027777
Other languages
French (fr)
Japanese (ja)
Inventor
惇一 清水
猛史 荻田
Original Assignee
ソニー株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニー株式会社 filed Critical ソニー株式会社
Publication of WO2021024753A1 publication Critical patent/WO2021024753A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer

Definitions

  • the present technology relates to an information processing device, an information processing method, and a program, for example, an information processing device, an information processing method, and a program capable of presenting vibrations related to an imaged object to a user.
  • Patent Document 1 there is known a technique for obtaining a desired perceptual effect by presenting a tactile sensation to a user by vibration stimulation or the like (see, for example, Patent Document 1).
  • the vibration can be presented. However, it is difficult to properly present the vibration associated with the state deviating from the predetermined state described above unless the data is prepared in advance.
  • This technology was made in view of such a situation, and makes it possible to present the vibration according to the state of the imaged object to the user.
  • the information processing device on one aspect of the present technology includes a recognition unit that recognizes an object in a captured image and a vibration data generation unit that generates vibration data according to the state of the object recognized by the recognition unit. Be prepared.
  • the information processing device recognizes an object in the captured image and generates vibration data according to the state of the recognized object.
  • the program of one aspect of the present technology causes a computer to perform a process including a step of recognizing an object in a captured image and generating vibration data according to the state of the recognized object.
  • an object in the captured image is recognized, and vibration data corresponding to the state of the recognized object is generated.
  • the information processing device may be an independent device or an internal block constituting one device.
  • the program can be provided by transmitting via a transmission medium or by recording on a recording medium.
  • this technique can be applied when the information processing device 10 presents vibrations corresponding to an image captured by a smartphone as the information processing device 10.
  • a smartphone will be taken as an example as the information processing device 10, but the present technology described below can be applied to other devices as well.
  • FIG. 2 is a block diagram showing a hardware configuration example of an embodiment of a smartphone as an information processing device 10 to which the present technology is applied.
  • the information processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a sensor unit 14, an input / output interface 15, a display unit 16, and a touch input unit. It has 17, a communication unit 18, a home button 19, and the like.
  • CPU Central Processing Unit
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the CPU 11 to the input / output interface 15 are connected to each other via a bus.
  • the display unit 16 to the home button 19 are connected to the input / output interface 15.
  • the CPU 11 performs various processes by executing a program stored in the ROM 12 or the RAM 13.
  • the ROM 12 and the RAM 13 store a program executed by the CPU 11 and data necessary for the operation of the CPU 11.
  • the sensor unit 14 includes an acceleration sensor, various optical sensors, and other necessary sensors, and senses a predetermined physical quantity.
  • the optical sensor include a monaural RGB camera that captures an RGB image, a monaural gray camera that captures a black-and-white (grescale) image, and a ranging sensor that obtains a distance image whose pixel value is a distance.
  • Distance measurement sensors include TOF sensors, structured light cameras, (patterned) stereo cameras, and the like.
  • the optical sensor can be provided on the display unit 16 side or the back surface side of the display unit 16.
  • the input / output interface 15 functions as an interface for the display unit 16 to the home button 19 and other devices.
  • the display unit 16 displays an image.
  • the touch input unit 17 accepts a user's touch operation (tap, flip, etc.).
  • the display unit 16 and the touch input unit 17 can be integrally configured by a touch panel or the like.
  • the communication unit 18 communicates by a predetermined communication method such as a wireless LAN (Local Area Network).
  • the home button 19 is a physical or virtual button provided at the bottom of the information processing device 10.
  • the oscillator 20 has a function of giving vibration to the information processing device 10.
  • the CPU 11 (computer) executes a program pre-installed in the ROM 12 or the RAM 13 or a program downloaded by the communication unit 18 and installed in the ROM 12 or the RAM 13. , Perform various processes. As a result, the information processing apparatus 10 presents the user with vibrations corresponding to the imaged object described below.
  • processing performed by the CPU 11 as a computer according to the program does not necessarily have to be performed in chronological order in the order described as the flowchart described later. That is, the processing performed by the computer according to the program also includes processing executed in parallel or individually (for example, parallel processing or processing by an object).
  • the program may be processed by one computer (processor) or may be distributed by a plurality of computers. Further, the program may be transferred to a distant computer and executed.
  • FIG. 3 is a diagram showing a functional configuration example of the information processing device 10.
  • the information processing device 10 includes an imaging unit 41, an object recognition unit 42, a vibration data acquisition unit 43, a vibration data storage unit 44, a difference determination unit 45, an attribute parameter acquisition unit 225, an attribute parameter storage unit 47, a correction processing unit 48, and A vibration generation presentation unit 49 is provided.
  • the imaging unit 41 is a camera including an image sensor and images an object.
  • the data of the object image captured by the image pickup unit 41 is supplied to the object recognition unit 42.
  • the object recognition unit 42 recognizes an object in the captured image.
  • the recognized object information (hereinafter, appropriately referred to as recognized object information) is supplied to the vibration data acquisition unit 43 and the difference determination unit 45.
  • the object recognition information is, for example, information such as a predetermined object and the number and amount of the objects.
  • the vibration data acquisition unit 43 is vibration data stored in the vibration data storage unit 44, and acquires vibration data based on the recognized object information.
  • the vibration data storage unit 44 stores an object and vibration data related to the object. In addition, the vibration data is stored in association with the number and quantity of objects.
  • the vibration data acquisition unit 43 reads the vibration data representing the vibration when the horse is running from the vibration data storage unit 44. Further, the vibration data read out is, for example, data stored in association with the number of objects such as one horse.
  • the vibration data acquired by the vibration data acquisition unit 43 is supplied to the difference determination unit 45.
  • the object recognition information from the object recognition unit 42 is also supplied to the difference determination unit 45.
  • the difference determination unit 45 determines the difference between the number and amount of objects represented by the object recognition information (hereinafter, the number will be described as an example) and the number of objects associated with the vibration data.
  • the difference determination unit 45 compares the state of the object represented by the object recognition information with the state of the object associated with the vibration data, and calculates the difference.
  • the object represented by the object recognition information is a horse
  • the number of the objects is two
  • the number associated with the vibration data is one
  • the information on the number of objects associated with the vibration data may be an image.
  • the object associated with the vibration data is an image captured, and the difference between the objects may be extracted by comparing the image with the captured image as the object recognition information.
  • the difference determination unit 45 supplies (transfers) the vibration data supplied from the vibration data acquisition unit 43 to the correction processing unit 48. Further, when the difference determination unit 45 determines that there is a difference, the difference determination unit 45 instructs the attribute parameter acquisition unit 225 to acquire the attribute parameter. The attribute parameter acquisition unit 225 acquires the attribute parameter from the attribute parameter storage unit 47.
  • the vibration data stored in the vibration data storage unit 44 is data related to vibration when the number of objects is a predetermined number.
  • the vibration data stored in the vibration data storage unit 44 will be appropriately referred to as reference vibration data.
  • the number and amount of objects associated with the reference vibration data are described as the reference number (quantity).
  • the parameter for converting the reference vibration data into the vibration data matching the number of objects represented by the object recognition information is the attribute parameter.
  • the correction processing unit 48 supplies the vibration data supplied via the difference determination unit 45 to the vibration generation presentation unit 49.
  • the correction processing unit 48 uses the vibration data supplied through the difference determination unit 45 as the attribute parameter supplied from the attribute parameter acquisition unit 225. It is corrected by using and supplied to the vibration generation presentation unit 49.
  • the vibration generation presentation unit 49 generates vibration based on the supplied vibration data and presents it to the user.
  • FIG. 4 shows a database configuration example of the vibration data storage unit 44 and the attribute parameter storage unit 47.
  • the vibration data storage unit 44 and the attribute parameter storage unit 47 are shown separately in FIG. 3 for explanation, they can be combined into one database as shown in FIG. 4, and in FIG. 4, one database is used.
  • An example of the configuration is shown below.
  • the vibration data storage unit 44 stores the object name, the action, the vibration data, and the attribute data in association with each other. Further, the attribute parameter storage unit 47 stores at least the object name and the attribute parameter in association with each other.
  • the object name is entered in the object name column.
  • the object name is information as an identifier that identifies the prepared vibration data. Although the object name is used here, it may be an image, sound, or the like of the object, or an ID assigned to a predetermined object.
  • the vibration data column if any action from the user is required when presenting vibration, that action is described.
  • Information on the presented vibration data (reference vibration data) is stored in the vibration data column.
  • the vibration data information includes vibration magnitude, pitch, time, and afterglow.
  • the reference image is an image in which a reference state of a predetermined object is captured.
  • the reference image is an image in which a predetermined container is filled with water, and such an image can be treated as an image in which an object and an amount of the object are associated with each other.
  • the reference vibration data is data for presenting the vibration when the number of objects is the number of objects described in the attribute data column.
  • the information for generating the vibration data when the number of objects other than the number of objects described in the attribute data column is other than the number of objects is the attribute parameter.
  • the object name is horse (stationary)
  • the action is blank
  • the vibration data is horse vibration data A
  • the attribute data is one horse is associated.
  • the reference vibration data when the horse 101 is stationary is the horse vibration data A.
  • the vibration presentation is started when, for example, an image of a stationary horse is captured, regardless of the user's action.
  • vibration data for the number of heads is generated.
  • the horse vibration data A contains 2 By multiplying, vibration data when there are two horses is generated and presented to the user by the information processing device 10.
  • the difference determination unit 45 compares the state where there are two stationary horses from the recognized object information and the state where there is one stationary horse from the attribute data, and compares the comparison result. , The difference judgment result.
  • the correction (generation) process as described above is executed according to the difference determination result.
  • an image of one stationary horse is stored as a reference image, for example, the image shown in FIG. 5A is stored as a reference image so that the difference from this reference image can be extracted. Is also good.
  • the image captured by the imaging unit 41 is an image of two horses that are stationary, the captured image and the reference image are compared, and the difference from the reference image, in this case, the difference of one horse. The difference between the existence and the presence may be extracted. Further, the vibration data may be corrected according to the extracted difference.
  • the object name is horse (sprinting)
  • the action is blank
  • the vibration data is horse vibration data B
  • the attribute data is one horse is associated.
  • the reference vibration data at the time of the running horse 111 is the horse vibration data B.
  • the vibration presentation is started when, for example, an image of a running horse is captured, regardless of the user's action.
  • vibration data for the number of heads is generated.
  • the horse vibration data B shows 2
  • vibration data when two horses are running is generated and presented to the user by the information processing device 10.
  • the difference determination unit 45 compares the state where there are two running horses from the recognized object information and the state where there is one running horse from the attribute data, and compares the comparison result. , The difference judgment result.
  • the correction (generation) process as described above is executed according to the difference determination result.
  • an image of one running horse is stored as a reference image, for example, the image shown in FIG. 6A is stored as a reference image so that the difference from this reference image can be extracted. Is also good.
  • the image captured by the imaging unit 41 is an image of two horses running, the captured image and the reference image are compared, and the difference from the reference image, in this case, the difference of one horse. The difference between the existence and the presence may be extracted. Further, the vibration data may be corrected according to the extracted difference.
  • the object name is water (shaking)
  • the action is vertical vibration
  • the vibration data is water vibration data A
  • the attribute data is 500 ml
  • these data are based on the water vibration data when the container 131 is shaken up and down when the container 131 such as a PET bottle is 500 ml (full state). It means that it is A.
  • the user takes an action such as vertical vibration, for example, shaking the information processing device 10 up and down, or moving the screen of the information processing device 10 up and down while touching it with a finger.
  • vertical vibration for example, shaking the information processing device 10 up and down, or moving the screen of the information processing device 10 up and down while touching it with a finger.
  • the vibration of water vibrating up and down in the container 131 is presented to the user.
  • vibration data matching the water amount is generated.
  • the information processing apparatus 10 images a container 132 containing 200 ml of water.
  • FIG. 8A when the user moves the display unit 16 (touch panel) up and down while touching it with a finger, the presentation of vibration is started.
  • vibration data of vibration when 200 ml of water moves up and down in the container 132 is generated by multiplying water vibration data A by (200 ⁇ a), and vibration based on the vibration data is generated. , Presented to the user.
  • the difference determination unit 45 compares the state where there is 200 ml of water from the recognized object information and the state where there is 500 ml of water from the attribute data, and sets the comparison result as the difference determination result.
  • the correction (generation) process as described above is executed according to the difference determination result.
  • an image of a container containing 500 ml of water is stored as a reference image, for example, an image as shown in A of FIG. 7 is stored as a reference image, and a difference from this reference image is extracted. You can do it.
  • the image captured by the imaging unit 41 is an image of a container containing 200 ml of water
  • the captured image is compared with the reference image, and the difference from the reference image, in this case, the difference of 300 ml less. May be extracted as a difference.
  • the vibration data may be corrected according to the extracted difference.
  • the object name is water (pouring)
  • the action is tilted
  • the vibration data is water vibration data B
  • the attribute data is 500 ml
  • these data are obtained when the container 131 such as a PET bottle is 500 ml (full state) and the container 131 is tilted (water is poured from the container 131 into a cup or the like). It means that the reference vibration data of) is the water vibration data B.
  • the water vibration data B is tilted as an action by the user, for example, tilting the information processing device 10 or moving the water vibration data B in a direction in which water comes out from the mouth of the container 131 while touching the screen of the information processing device 10 with a finger.
  • an action such as is performed, the vibration when water is poured from the container 131 into a cup or the like is presented to the user.
  • vibration data matching the water amount is generated.
  • the information processing apparatus 10 images a container 132 containing 200 ml of water.
  • the presentation of vibration is started.
  • the vibration data of the vibration when 200 ml of water is poured from the container 132 is generated by multiplying the water vibration data B by (200 ⁇ b), and the vibration based on the vibration data is generated.
  • the user tilts the information processing device 10.
  • the object name is a car (idling)
  • the action is a tap
  • the vibration data is the vehicle vibration data A
  • the attribute data is one is associated.
  • the difference determination unit 45 compares the state where there are two idling cars from the recognized object information and the state where there is one idling car from the attribute data, and compares them. The result is used as the difference determination result.
  • the correction (generation) process as described above is executed according to the difference determination result.
  • an image of one idling car is stored as a reference image, for example, the image shown in FIG. 9A is stored as a reference image so that the difference from this reference image can be extracted. Is also good.
  • the image captured by the imaging unit 41 is an image of two idling cars
  • the captured image is compared with the reference image, and the difference from the reference image, in this case, the difference of one car.
  • the difference between the existence and the presence may be extracted. Further, the vibration data may be corrected according to the extracted difference.
  • vibration data for the number of units is generated.
  • the vehicle vibration data A contains 2 By multiplying, vibration data when there are two cars is generated and presented to the user by the information processing device 10.
  • the reference vibration data and the parameters for correcting the reference vibration data are stored in the information processing apparatus 10.
  • the action may be approaching an object (the imaged object becomes larger, it can be determined from the position data that the object is approaching, etc.), tracing, etc., or an instruction is given by voice from the user. It may be when there is. Further, as an action, the force of pressing the touch panel may be taken into consideration.
  • vibrations may be presented depending on the type of action. For example, as an action of a car (idling) as an object name shown in FIG. 4, the above processing is performed in the case of tapping, and vibration when the car is running is presented in the case of tracing (sliding). The processing in which the attribute parameter is selected may be performed.
  • the information processing device 10 stores the reference vibration data and parameters for correcting the reference voice data. Further, as information on the reference vibration data, information on the state of the object including the number and quantity of the objects is stored, the difference from the information is extracted, and the information is converted into vibration data according to the difference. Therefore, it is possible to present the user with vibration according to the state of the object.
  • an image relating to the state of the object including the number and quantity of the objects is prepared as a reference image, and the difference between the reference image and the captured image is extracted and corresponding to the difference. Converted to vibration data. Therefore, it is possible to present the user with vibration according to the state of the object.
  • step S11 the image is captured by the imaging unit 41.
  • the user points the camera at a predetermined object while holding the information processing device 10, and imaging is performed.
  • step S12 the object recognition unit 42 analyzes the image captured by the image pickup unit 41, detects the object in the image, the number, amount, state, etc. of the object, and uses the vibration data acquisition unit 43 as the recognition object information. Is supplied to the difference determination unit 45.
  • step S13 the vibration data acquisition unit 43 reads the data associated with the object (object name) indicated by the recognized object information from the vibration data storage unit 44.
  • the data to be read is action, vibration data, and attribute data as described with reference to FIG.
  • the read data is supplied to the difference determination unit 45.
  • step S14 the difference determination unit 45 determines whether or not there is a difference in the object information.
  • the difference determination unit 45 refers to the attribute data supplied from the vibration data acquisition unit 43 and the recognized object information supplied from the object recognition unit 42, and indicates the number (quantity) indicated by the attribute data and the recognized object information. Compare the number (quantity) to be obtained and determine whether or not there is a difference.
  • the number indicated by the perceptible object information is 2, and the number indicated by the attribute data is 1, so that there is a difference. Is issued. Further, when one horse is photographed, the number indicated by the recognized object information is 1, and the number indicated by the attribute data is 1, so it is determined that there is no difference.
  • the amount of water to be collected may be estimated to be about 2/3 or 1/2.
  • the difference may be calculated by comparing the images.
  • a reference image may be stored as attribute data, and the difference from the reference image may be extracted.
  • a reference image an image of a container containing 500 ml of water is prepared, and the reference image is compared with the image of the container containing 200 ml of water taken, and the difference from the reference image, in this case, , The difference that the amount of 300 ml is different may be extracted. Then, the vibration data is corrected according to the extracted difference.
  • the type of data stored as attribute data may differ depending on the object.
  • the attribute data of an object whose number can be counted as an object may be used as information on the number of objects, and the attribute data of an object whose quantity can be compared may be used as a reference image.
  • step S14 determines in step S14 that there is a difference
  • the process proceeds to step S15.
  • the difference determination unit 45 determines that there is a difference
  • the difference determination unit 45 supplies the information that there is a difference and the vibration data to the correction processing unit 48, and instructs the attribute parameter acquisition unit 225 to acquire the attribute parameter.
  • step S15 the attribute parameter acquisition unit 225 acquires the corresponding attribute parameter from the attribute parameter storage unit 47. For example, when the vibration data associated with the horse (stationary) has been acquired, the attribute parameter associated with the horse (stationary) is also acquired as the attribute parameter. The acquired attribute parameter is supplied to the correction processing unit 48.
  • step S16 the correction processing unit 48 corrects the vibration data using the attribute parameters. For example, as described with reference to FIG. 5, when two horses are photographed, a parameter of "number of horses times" is acquired as an attribute parameter, so that the correction processing unit 48 performs horse vibration data A. Is doubled.
  • the correction processing unit 48 corrects the water vibration data A by (200 ⁇ a) times.
  • two data, water (shaking) and water (pouring), are stored in the database as data related to water.
  • these two data are also associated with an action, and vibration is started when there is a user action. Therefore, which of the two data, water (shaking) and water (pouring), cannot be read at the time when the container containing water is photographed.
  • vibration data In the case of such data, two data may be read out, corrected for each, and vibration data may be generated.
  • the order of processing according to the flow chart shown in FIG. 10 should be changed, and after the user has taken an action, that is, which of the two data, water (shaking) and water (pouring), should be read.
  • the corresponding data When is determined, the corresponding data may be read out, corrected, and vibration data may be generated.
  • step S16 When the vibration data is corrected by the correction processing unit 48 in step S16, the corrected vibration data is supplied to the vibration generation presentation unit 49, and the processing proceeds to step S17.
  • step S17 also comes when it is determined in step S14 that there is no difference in the object information.
  • the difference determination unit 45 determines that there is no difference, the difference determination unit 45 supplies the information that there is no difference and the vibration data to the correction processing unit 48.
  • the correction processing unit 48 receives the information that there is no difference, the correction processing unit 48 supplies the vibration data to the vibration generation presentation unit 49.
  • step S17 the vibration generation presentation unit 49 determines whether or not the object requires an action from the user. This determination is made by determining whether or not there is data described as an action.
  • step S17 If it is determined in step S17 that the object requires an action from the user, the process proceeds to step S18.
  • step S18 the vibration generation presentation unit 49 determines whether or not there has been an action from the user.
  • the vibration generation presentation unit 49 holds the vibration data of water (shaking) and the vibration data of water (pouring)
  • the vibration data of water (shaking) is vibrated up and down as an action. It is set to be presented at times, and the vibration data of water (pouring) is set to be presented when the action of tilting is performed as an action. In such a case, it is determined whether or not an operation such as vertical vibration or tilting has been performed.
  • the vibration generation presentation unit 49 determines that there is an action from the user, and proceeds to the process in step S19.
  • step S19 the vibration generation presentation unit 49 generates vibration based on the vibration data and presents it to the user. For example, when the user shakes the information processing device 10 in the vertical direction, vibration is generated based on the vibration data of water (shaking) and presented to the user.
  • step S17 if it is determined in step S17 that the object does not require an action from the user, the process proceeds to step S19.
  • the vibration generation presentation unit 49 holds the vibration data of the horse (stationary)
  • the vibration data of the horse (stationary) is an object that requires an action from the user because the action is not specified. It is judged that it is not.
  • the vibration related to the horse is generated from the vibration data of the horse (stationary) and presented to the user.
  • vibration related to the imaged object is presented to the user, so that the user can experience the vibration of the object in a simulated manner.
  • vibration can be presented at the time of imaging, or vibration can be presented at the time when some action related to the object is performed. Even if it is the same object, it is possible to present the vibration according to the action to the user.
  • a part of the processing performed by the information processing device 10 described above may be performed by the server.
  • the above-mentioned processing of presenting vibration to the user can be executed. Since the above description using the information processing device 10 as an example can be applied to the information processing system 200 described below, duplicate description will be omitted as appropriate, but even if it is not described below, it may be omitted. The above description will continue as if it were applicable as appropriate.
  • the information processing device 201 shown in FIG. 11 includes an imaging unit 211, an object recognition unit 212, a communication unit 213, and a vibration generation presentation unit 214.
  • the server 202 includes a communication unit 221, a voice data acquisition unit 222, a voice data storage unit 223, a difference determination unit 224, an attribute parameter acquisition unit 225, an attribute parameter storage unit 226, and a vibration data generation unit 227.
  • the imaging unit 211 of the information processing device 201 is configured to include a camera that images an object.
  • the data of the object image captured by the image pickup unit 211 is supplied to the object recognition unit 212.
  • the object recognition unit 212 recognizes an object in the captured image.
  • the recognized object information (recognized object information) is supplied to the communication unit 213.
  • the object recognition information is, for example, information such as a predetermined object and the number and amount of the objects.
  • the communication unit 213 communicates with the server 202 and transmits the object recognition information to the server 202. Further, the communication unit 213 receives the vibration data from the server 202 and supplies it to the vibration generation presentation unit 214. The vibration generation presentation unit 214 generates vibration based on the supplied vibration data and presents it to the user.
  • the object recognition unit 212 may be configured to have the server 202. Further, in such a configuration, the image captured by the imaging unit 211 is transmitted to the server 202 via the communication unit 213.
  • the communication unit 221 of the server 202 communicates with the information processing device 201 and receives the object recognition information. Further, the communication unit 221 transmits the vibration data to the information processing device 201.
  • the object recognition information received by the communication unit 221 is supplied to the voice data acquisition unit 222 and the difference determination unit 224.
  • the voice data acquisition unit 222 is provided in the server 202.
  • the voice data acquisition unit 222 is voice data stored in the voice data storage unit 223, and acquires voice data based on the recognized object information.
  • the voice data storage unit 223 stores an object and voice data related to the object. Further, the voice data is stored in association with the number and quantity of objects.
  • the voice data acquisition unit 222 transmits voice data representing a sound (for example, footsteps or barks) when the horse is running from the voice data storage unit 223. read out. Further, the read voice data is, for example, data stored in association with the number of objects such as one horse.
  • the voice data acquired by the voice data acquisition unit 222 is supplied to the difference determination unit 224.
  • Object recognition information from the information processing device 201 is also supplied to the difference determination unit 224 via the communication unit 221.
  • the difference determination unit 224 determines the difference between the number and amount of objects represented by the object recognition information (hereinafter, the number will be described as an example) and the number of objects associated with the voice data.
  • the object represented by the object recognition information is a horse
  • the number of the objects is two
  • the number associated with the voice data is one
  • the difference determination unit 224 supplies (transfers) the voice data supplied from the voice data acquisition unit 222 to the vibration data generation unit 227. Further, when the difference determination unit 224 determines that there is a difference, the difference determination unit 224 instructs the attribute parameter acquisition unit 225 to acquire the attribute parameter. The attribute parameter acquisition unit 225 acquires the attribute parameter from the attribute parameter storage unit 226.
  • the voice data stored in the voice data storage unit 223 is data related to voice when the number of objects is a predetermined number.
  • the voice data stored in the voice data storage unit 223 will be appropriately referred to as reference voice data. Further, the number and amount of objects associated with the reference voice data are described as the reference number (quantity).
  • the parameter for converting the reference voice data into the voice data matching the number of objects represented by the object recognition information is the attribute parameter.
  • the vibration data generation unit 227 converts the voice data supplied via the difference determination unit 224 into vibration data.
  • the vibration data generation unit 227 uses the audio data supplied via the difference determination unit 224 as the attribute parameter supplied from the attribute parameter acquisition unit 225. Is corrected using, and the corrected audio data is converted into vibration data.
  • the vibration data generated by the vibration data generation unit 227 is supplied to the communication unit 221.
  • the communication unit 221 transmits the supplied vibration data to the information processing device 201.
  • vibration data is generated from voice data.
  • the information processing system 200 shown in FIG. 11 can also be configured to store vibration data and correct the vibration data as necessary, as in the information processing device 10 shown in FIG.
  • the audio data can be acquired from, for example, a video on the Internet. By acquiring from the video, it is possible to acquire a predetermined object and the sound of the object. As described above, it is considered that the object and the audio data about the object are more present than the vibration data about the object and the object, and the audio data can be used in the information processing system 200.
  • the information processing device 10 (FIG. 3) is configured to store vibration data, or when the information processing device 10 is a terminal such as a smartphone, a large amount of vibration data is stored because the storage capacity is limited. It may not be possible. However, as in the information processing system 200, by providing the server 202 and configuring the server 202 side to store the voice data for generating the vibration data, more voice data can be stored. Therefore, it becomes possible to supply more vibration data.
  • FIG. 12 shows a database configuration example of the voice data storage unit 223 and the attribute parameter storage unit 226.
  • the voice data storage unit 223 and the attribute parameter storage unit 226 are shown separately in FIG. 11 for the sake of explanation, they can be combined into one database as shown in FIG. 12, and in FIG. 12, one database is used.
  • An example of the configuration is shown below.
  • the voice data storage unit 223 stores the object name, the action, the voice data, and the attribute data in association with each other. Further, the attribute parameter storage unit 226 stores at least the object name and the attribute parameter in association with each other.
  • the object name is entered in the object name column.
  • the action column if any action is required from the user when presenting the vibration, that action is described.
  • Information on the presented voice data (reference voice data) is stored in the voice data column.
  • Information on voice data includes loudness, frequency, pitch, time, and afterglow.
  • attribute data column data on the number and amount of objects and an image (reference image) in which the state of the reference object is captured are described.
  • attribute parameter column of the attribute parameter storage unit 2266 information for correcting the voice data is described when the number or amount is other than the number or amount of objects described in the attribute data.
  • the reference voice data is data for generating voice when the number of objects is the number of objects described in the attribute data column.
  • the information for generating the voice data when the number of objects is other than the number of objects described in the attribute data column is the attribute parameter. Vibration data is generated from this corrected voice data.
  • the object name is horse (stationary), the action is blank, the voice data is horse voice data A, and the attribute data is one horse is associated.
  • the reference voice data when the horse 101 is stationary is the horse voice data A.
  • the horse voice data A is converted into vibration data when, for example, an image of a stationary horse is captured, regardless of the user's action, and the presentation of vibration is started.
  • voice data for the number of heads is generated.
  • the horse voice data A contains 1
  • voice data when there are two horses is generated, the voice data is converted into vibration data, and the information processing device 10 presents vibration to the user.
  • the object name is a horse (sprinting)
  • the action is blank
  • the voice data is horse voice data B
  • the attribute data is one is associated.
  • These data mean that, as shown in A of FIG. 6, the reference voice data at the time of the running horse 111 is the horse voice data B.
  • the horse voice data B is converted into vibration data when, for example, an image of a running horse is captured, regardless of the user's action, and the presentation of vibration is started.
  • the information of double is described.
  • voice data for the number of heads is generated.
  • the horse voice data B contains 2
  • voice data when two horses are running is generated, the corrected voice data is converted into vibration data, and the information processing device 10 presents vibration to the user.
  • information that the object name is water (shaking), the action is vertical vibration, the voice data is water voice data A, and the attribute data is 500 ml is associated. As shown in A of FIG. 7, these data are based on the water voice data when the container 131 is shaken up and down when the container 131 such as a PET bottle is 500 ml (full state). It means that it is A.
  • the user takes an action such as vertical vibration, for example, shaking the information processing device 10 up and down, or moving the screen of the information processing device 10 up and down while touching the screen with a finger.
  • vertical vibration for example, shaking the information processing device 10 up and down, or moving the screen of the information processing device 10 up and down while touching the screen with a finger.
  • it is sound data when water vibrates up and down in the container 131. Then, the water voice data A is converted into vibration data, so that the vibration is presented to the user.
  • the information processing apparatus 10 images a container 132 containing 200 ml of water.
  • the display unit 16 touch panel
  • the presentation of vibration is started.
  • the sound data when 200 ml of water moves up and down in the container 132 is generated by correcting to increase the high-pitched part of the water sound data A, and the vibration converted based on the sound data.
  • the vibration based on the data is presented to the user.
  • the object name is water (pouring)
  • the action is tilted
  • the voice data is water voice data B
  • the attribute data is 500 ml
  • these data are obtained when the container 131 such as a PET bottle is 500 ml (full state) and the container 131 is tilted (water is poured from the container 131 into a cup or the like). It means that the reference audio data of (when) is the water audio data B.
  • the water voice data B can be tilted as an action by the user, for example, tilting the information processing device 10 or moving the water voice data B in a direction in which water comes out from the mouth of the container 131 while touching the screen of the information processing device 10 with a finger.
  • This is sound data when water is poured from a container 131 into a cup or the like when an action such as is performed.
  • the water voice data B is converted into vibration data, so that the vibration is presented to the user.
  • the information processing apparatus 10 images a container 132 containing 200 ml of water. Then, as shown in FIG. 8B, when the user tilts the information processing device 10, the presentation of vibration is started.
  • the vibration data of the sound when 200 ml of water is poured from the container 132 is generated by correcting to increase the treble region of the water voice data B, and the vibration data is converted based on the voice data. The vibration based on is presented to the user.
  • these data mean that, as shown in A of FIG. 9, the reference voice data when the vehicle is idling 151 is the car voice data A. Further, the vehicle voice data A is converted into vibration data, and when the user taps the information processing device 10, the presentation of vibration based on the vibration data is started.
  • the volume of the reference voice data by doubling the volume of the reference voice data, the voice data at the time of idling is generated, and by doubling the volume, the voice data at the time of traveling is generated.
  • the volume of the car voice data A is set.
  • the reference voice data and the parameters for correcting the reference voice data are stored in the server 202.
  • information on reference audio data information on the state of an object including the number and quantity of objects is stored, a difference from the information is extracted, converted into audio data according to the difference, and the audio is further generated. The data is converted to vibration data. Therefore, it is possible to present the user with vibration according to the state of the object.
  • an image relating to the state of the object including the number and quantity of the objects is prepared as the reference image, and the difference between the reference image and the captured image is extracted, and according to the difference. It is converted into audio data, and the audio data is further converted into vibration data. Therefore, it is possible to present the user with vibration according to the state of the object.
  • step S101 an image is captured by the imaging unit 211 of the information processing device 201. While the user holds the information processing device 201, the user points the camera at a predetermined object to perform imaging.
  • step S102 the object recognition unit 212 analyzes the image captured by the image pickup unit 211, detects an object in the image, the number, amount, state, etc. of the object, and generates object recognition information including the information. To do.
  • step S103 the recognized object information generated by the object recognition unit 212 is transmitted to the server 202 via the communication unit 213.
  • step S121 the communication unit 221 of the server 202 receives the recognized object information from the information processing device 201.
  • the received recognized object information is supplied to the voice data acquisition unit 222 and the difference determination unit 224.
  • step S122 the voice data acquisition unit 222 reads the data associated with the object (object name) indicated by the recognized object information from the voice data storage unit 223.
  • the data to be read is action, voice data, and attribute data, as described with reference to FIG.
  • the read data is supplied to the difference determination unit 224.
  • step S123 the difference determination unit 224 determines whether or not there is a difference in the object information.
  • the difference determination unit 224 refers to the attribute data supplied from the voice data acquisition unit 222 and the recognized object information supplied from the object recognition unit 212, and indicates the number (quantity) indicated by the attribute data and the recognized object information. Compare the number (quantity) of the data and determine if there is a difference.
  • the number indicated by the perceptible object information is 2, and the number indicated by the attribute data is 1, so that there is a difference. Is issued. Further, when one horse is photographed, the number indicated by the recognized object information is 1, and the number indicated by the attribute data is 1, so it is determined that there is no difference.
  • step S123 If the difference determination unit 224 determines in step S123 that there is a difference, the process proceeds to step S124.
  • the difference determination unit 224 determines that there is a difference, the difference determination unit 224 supplies the information that there is a difference and the voice data to the vibration data generation unit 227, and instructs the attribute parameter acquisition unit 225 to acquire the attribute parameter.
  • step S124 the attribute parameter acquisition unit 225 acquires the corresponding attribute parameter from the attribute parameter storage unit 226. For example, when the voice data associated with the horse (stationary) has been acquired, the attribute parameter associated with the horse (stationary) is also acquired as the attribute parameter. The acquired attribute parameters are supplied to the vibration data generation unit 227.
  • the vibration data generation unit 227 corrects the voice data using the attribute parameters. For example, as described with reference to FIG. 5, when two horses are photographed, a parameter of “volume 1.3 times” is acquired as an attribute parameter, so that the vibration data generation unit 227 may use the vibration data generation unit 227. The volume of the horse voice data A is corrected by 1.3 times.
  • a parameter of “treble increase” is acquired as an attribute parameter, so that the vibration data generation unit At 227, correction is performed to increase the treble region of the water voice data A.
  • two data, water (shaking) and water (pouring), are stored in the database as data related to water.
  • these two data are also associated with an action, and vibration is started when there is a user action. Therefore, which of the two data, water (shaking) and water (pouring), cannot be read at the time when the perceptible object information is acquired.
  • two voice data may be read out, corrected for each, vibration data generated, and transmitted.
  • the order of processing according to the flow chart shown in FIG. 13 is changed so that when there is an action of the user, information about the action is transmitted from the information processing apparatus 201 side, and the transmitted information is transmitted.
  • the corresponding audio data is read, corrected, and vibrated. Data may be generated.
  • step S125 the vibration data generation unit 227 corrects the voice data. Then, in step S126, the vibration data generation unit 227 converts the voice data into vibration data and supplies it to the communication unit 221. The communication unit 221 transmits the supplied vibration data to the information processing device 201 in step S127.
  • step S126 also comes when it is determined in step S123 that there is no difference in the object information.
  • the difference determination unit 224 determines that there is no difference, the difference determination unit 224 supplies the information that there is no difference and the voice data to the vibration data generation unit 227.
  • the vibration data generation unit 227 receives the information that there is no difference, the vibration data generation unit 227 converts the voice data into vibration data and supplies it to the communication unit 221.
  • the communication unit 221 transmits the supplied vibration data to the information processing device 201 in step S127.
  • the vibration data transmitted from the server 202 is received by the communication unit 213 of the information processing device 201 in step S104.
  • the received vibration data is supplied to the vibration generation presentation unit 214.
  • the vibration generation presentation unit 214 determines whether or not the object requires an action from the user. This determination is made by determining whether or not there is data set as an action. Further, the data for making such a determination is also transmitted from the server 202 side together with the vibration data.
  • step S105 If it is determined in step S105 that the object requires an action from the user, the process proceeds to step S106.
  • step S106 the vibration generation presentation unit 214 determines whether or not there has been an action from the user.
  • the vibration generation presentation unit 214 holds the vibration data of water (shaking) and the vibration data of water (pouring), the vibration data of water (shaking) is vibrated up and down as an action. It is set to be presented at times, and the vibration data of water (pouring) is set to be presented when the action of tilting is performed as an action. In such a case, it is determined whether or not an operation such as vertical vibration or tilting has been performed.
  • the vibration generation presentation unit 214 determines that there is an action from the user, and proceeds to the process in step S107.
  • step S107 the vibration generation presentation unit 214 generates vibration based on the vibration data and presents it to the user. For example, when the user shakes the information processing device 201 in the vertical direction, vibration is generated based on the voice data of water (shaking) and presented to the user.
  • step S105 if it is determined in step S105 that the object does not require an action from the user, the process proceeds to step S107.
  • the vibration generation presentation unit 214 holds the voice data of the horse (stationary)
  • the voice data of the horse (stationary) is an object that requires an action from the user because the action is not specified. It is judged that it is not.
  • the vibration related to the horse is generated from the vibration data of the horse (stationary) and presented to the user.
  • vibration related to the imaged object is presented to the user, so that the user can experience the vibration of the object in a simulated manner.
  • vibration can be presented at the time of imaging, or vibration can be presented at the time when some action related to the object is performed. Even if it is the same object, it is possible to present the vibration according to the action to the user.
  • the information processing device 10 and the information processing system 200 described above present vibrations to the user, and a mechanism may be provided in which the user's evaluation is supplied as feedback with respect to the presented vibrations. Further, when a feedback mechanism is provided, the vibration data may be corrected according to the content of the feedback, and the vibration more suitable for the user's preference may be presented.
  • voice data is converted into vibration data, and this conversion may be learned so that vibration that is more suitable for the user's preference is presented.
  • Feedback can be feedback that defines vibration errors.
  • a GUI (Graphical User Interface) button for inputting an evaluation of good or bad for vibration is displayed on the information processing device 10 (201), and the information (good or bad) of the operated button is used as feedback. .. Further, for example, if the operation is interrupted while the user is shaking, it is determined that the presented vibration is not the user's preference, and the determination result can be used as feedback. good.
  • GUI Graphic User Interface
  • vibration is set to be presented when the object is being imaged, and if the user stops imaging the object while the vibration is being presented, the presented vibration will be presented by the user. It may be determined that the object is not preferred, and such a determination result may be fed back.
  • the feedback may be feedback that corrects the feeling of vibration.
  • a GUI button for inputting an evaluation such as strong or weak vibration is displayed on the information processing device 10 (201), and the information (strong or weak) of the operated button is used as feedback. Further, for example, a specific comment from the user regarding vibration may be used as feedback.
  • a setting is provided to weaken the vibration.
  • the terminal information of the information processing device 10 (201) during vibration may be acquired as feedback. For example, if the user obtains terminal information that the terminal has been moved away from an object during vibration, the vibration was strong, so it is determined that the terminal has been moved away, and the vibration that is slightly weakened will be presented from the next time. To.
  • the correction by feedback is vibration data in the case of the information processing device 10 shown in FIG. 3, and voice data in the case of the information processing system 200 shown in FIG.
  • the server 202 that constitutes the information processing system 200 can be configured as shown in FIG.
  • the server 202 shown in FIG. 14 is composed of three servers 202-1, server 202-2, and server 202-3.
  • the server 202-1 is supplied with object recognition information from the information processing device 201, and requests the server 202-2 for voice data regarding the object represented by the object recognition information.
  • the server 202-2 is a server that stores voice data, reads voice data in response to a request from server 202-1, and supplies the voice data to server 202-3.
  • the server 202-3 is a server that converts voice data into vibration data, and when voice data is supplied from server 202-2, the voice data is converted into vibration data and supplied to the information processing device 201.
  • the server 202-1 and the server 202-3 can be unique API (Application Programming Interface) servers.
  • the server 202-2 can be a public API server.
  • the server 202-2 stores audio data, and this storage can be performed by analyzing the video published on the net as described above and acquiring the object and the audio. Therefore, even if a server that has already been constructed is used as the server that stores voice data, processing to which the present technology is applied can be performed.
  • the server 202-1 and the server 202-3 are servers that perform processing for presenting vibration, and may be one server.
  • Feedback is supplied from the information processing device 201 to the server 202-1.
  • the server 202-1 transmits the content of the feedback and the perceptible object information (object name, etc.) that is the source of the feedback via the server 202-2 or directly to the server 202-3. Supply to.
  • the server 202-3 learns a conversion method when converting from voice data to vibration data by using the feedback, the perceptible object information that is the source of the feedback, and the voice data that is the basis of the feedback. For example, using reinforcement learning, when the recognized object information is used as input information and the vibration data is used as output information, learning is performed using the reward for the vibration data as feedback.
  • the information processing device 201 and the plurality of servers 202 can be configured to perform processing.
  • the information processing device 10-1 and the information processing device 10-2 image the same object (here, the ball 301).
  • the vibration related to the ball 301 is presented by the information processing device 10-1 and the information processing device 10-2. It is assumed that the vibration data of the ball 301 is associated with an action of "presenting vibration when another terminal is moved", for example.
  • step S17 When the information processing device 10-1 and the information processing device 10-2 execute the process based on the flowchart shown in FIG. 10, when determining in step S17 whether or not the object requires an action from the user. Since the above-mentioned action is set, it is determined as YES, and the process proceeds to step S18.
  • step S18 it is determined whether or not there is an action from the user. In this case, since it is an action of "presenting vibration when another terminal is moved", the other terminal (state of the other party). When there is a change in the other party's condition, the vibration presentation is started.
  • the information processing device 10-1 when the information processing device 10-1 is moved, the information processing device 10-1 transmits the information that the information processing device 10-1 has moved to the information processing device 10-2.
  • the information processing device 10-2 receives the information that it has moved from the information processing device 10-1, the information processing device 10-2 starts to present the vibration corresponding to the information.
  • the vibration presented to the user can be a vibration according to the motion, the speed of movement, the distance, the numerical value on the screen, the time, the situation of the other party, and the like.
  • the terminals that are imaging the same object can be made to operate in conjunction with each other.
  • one action can cause the other terminal to initiate vibration, or both terminals to present the same vibration.
  • the server 202 executes processing for a plurality of information processing devices 201.
  • various vibrations can be presented to the user.
  • the series of processes described above can be executed by hardware or by software.
  • the programs constituting the software are installed on the computer.
  • the computer includes a computer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.
  • FIG. 16 is a block diagram showing a configuration example of computer hardware that executes the above-mentioned series of processes programmatically.
  • a CPU Central Processing Unit
  • ROM Read Only Memory
  • RAM Random Access Memory
  • An input / output interface 1005 is further connected to the bus 1004.
  • An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.
  • the input unit 1006 includes a keyboard, a mouse, a microphone, and the like.
  • the output unit 1007 includes a display, a speaker, and the like.
  • the storage unit 1008 includes a hard disk, a non-volatile memory, and the like.
  • the communication unit 1009 includes a network interface and the like.
  • the drive 1010 drives a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
  • the CPU 1001 loads the program stored in the storage unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes the above-described series. Is processed.
  • the program executed by the computer can be recorded and provided on the removable media 1011 as a package media or the like, for example. Programs can also be provided via wired or wireless transmission media such as local area networks, the Internet, and digital satellite broadcasting.
  • the program can be installed in the storage unit 1008 via the input / output interface 1005 by mounting the removable media 1011 in the drive 1010. Further, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. In addition, the program can be pre-installed in the ROM 1002 or the storage unit 1008.
  • the program executed by the computer may be a program that is processed in chronological order in the order described in this specification, or may be a program that is processed in parallel or at a necessary timing such as when a call is made. It may be a program in which processing is performed.
  • system represents the entire device composed of a plurality of devices.
  • the present technology can also have the following configurations.
  • a recognition unit that recognizes objects in the captured image
  • An information processing device including a vibration data generation unit that generates vibration data according to the state of the object recognized by the recognition unit.
  • the state of the object is a difference from the reference image.
  • the information processing device according to (1) above wherein the vibration data generation unit corrects according to the difference.
  • the state of the object is a difference between the number or amount of the object recognized by the recognition unit and the preset number or amount of the object.
  • the information processing apparatus according to (1) or (2), wherein the vibration data generation unit corrects according to the difference.
  • the information processing device according to any one of (1) to (9) above, wherein the vibration based on the vibration data is presented when there is an action of the user.
  • the information processing device Recognize objects in the captured image and An information processing method that generates vibration data according to the recognized state of the object.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The present technology relates to an information processing device, an information processing method, and a program which enable vibration corresponding to an object to be presented to a user. The present technology is provided with: a recognition unit which recognizes objects in a captured image; and a vibration data generation unit which generates vibration data corresponding to the state of the objects recognized by the recognition unit. The state of the objects indicates a difference from a reference image, and the vibration data generation unit performs correction according to the difference. The state of the object indicates the difference between the number or quantity of the objects recognized by the recognition unit and the preset number or quantity of objects, and the vibration data generation unit performs the correction corresponding to the difference. The present technology can be applied to, for example, an information processing device which presents vibration to a user.

Description

情報処理装置、情報処理方法、並びにプログラムInformation processing equipment, information processing methods, and programs
　本技術は情報処理装置、情報処理方法、並びにプログラムに関し、例えば、撮像された物体に関する振動をユーザに提示することができるようにした情報処理装置、情報処理方法、並びにプログラムに関する。 The present technology relates to an information processing device, an information processing method, and a program, for example, an information processing device, an information processing method, and a program capable of presenting vibrations related to an imaged object to a user.
　従来、振動刺激等による触覚提示をユーザに対して行うことで、所望の知覚効果が得られるようにする技術が知られている（例えば、特許文献１参照）。 Conventionally, there is known a technique for obtaining a desired perceptual effect by presenting a tactile sensation to a user by vibration stimulation or the like (see, for example, Patent Document 1).
特表2010-506302号公報Special Table 2010-506302
　所定の物体の所定の状態に対応付けられた振動を提示するためのデータが、予め用意されていれば、その振動を提示することはできる。しかしながら、前記した所定の状態から外れた状態に対応付けられた振動を提示することは、予めデータが用意されていなければ適切に提示することは困難である。 If the data for presenting the vibration associated with the predetermined state of the predetermined object is prepared in advance, the vibration can be presented. However, it is difficult to properly present the vibration associated with the state deviating from the predetermined state described above unless the data is prepared in advance.
　本技術は、このような状況に鑑みてなされたものであり、撮像された物体の状態に応じた振動が、ユーザに提示されるようにすることができるようにするものである。 This technology was made in view of such a situation, and makes it possible to present the vibration according to the state of the imaged object to the user.
　本技術の一側面の情報処理装置は、撮影された画像内の物体を認識する認識部と、前記認識部により認識された前記物体の状態に応じた振動データを生成する振動データ生成部とを備える。 The information processing device on one aspect of the present technology includes a recognition unit that recognizes an object in a captured image and a vibration data generation unit that generates vibration data according to the state of the object recognized by the recognition unit. Be prepared.
　本技術の一側面の情報処理方法は、情報処理装置が、撮影された画像内の物体を認識し、認識された前記物体の状態に応じた振動データを生成する。 In the information processing method of one aspect of the present technology, the information processing device recognizes an object in the captured image and generates vibration data according to the state of the recognized object.
　本技術の一側面のプログラムは、コンピュータに、撮影された画像内の物体を認識し、認識された前記物体の状態に応じた振動データを生成するステップを含む処理を実行させる。 The program of one aspect of the present technology causes a computer to perform a process including a step of recognizing an object in a captured image and generating vibration data according to the state of the recognized object.
　本技術の一側面の情報処理装置、情報処理方法、並びにプログラムにおいては、撮影された画像内の物体が認識され、認識された物体の状態に応じた振動データが生成される。 In the information processing device, information processing method, and program of one aspect of the present technology, an object in the captured image is recognized, and vibration data corresponding to the state of the recognized object is generated.
　なお、情報処理装置は、独立した装置であっても良いし、１つの装置を構成している内部ブロックであっても良い。 The information processing device may be an independent device or an internal block constituting one device.
　また、プログラムは、伝送媒体を介して伝送することにより、または、記録媒体に記録して、提供することができる。 Further, the program can be provided by transmitting via a transmission medium or by recording on a recording medium.
本技術を適用した情報処理装置の一例を示す図である。It is a figure which shows an example of the information processing apparatus to which this technology is applied. 情報処理装置のハードウエア構成を示す図である。It is a figure which shows the hardware configuration of an information processing apparatus. 情報処理装置の機能構成を示す図である。It is a figure which shows the functional structure of an information processing apparatus. データベースの構成を示す図である。It is a figure which shows the structure of a database. 撮像された画像と提示される振動について説明するための図である。It is a figure for demonstrating the captured image and the vibration presented. 撮像された画像と提示される振動について説明するための図である。It is a figure for demonstrating the captured image and the vibration presented. 撮像された画像と提示される振動について説明するための図である。It is a figure for demonstrating the captured image and the vibration presented. 撮像された画像と提示される振動について説明するための図である。It is a figure for demonstrating the captured image and the vibration presented. 撮像された画像と提示される振動について説明するための図である。It is a figure for demonstrating the captured image and the vibration presented. 情報処理装置の動作について説明するためのフローチャートである。It is a flowchart for demonstrating operation of an information processing apparatus. 情報処理システムの一実施の形態の構成を示す図である。It is a figure which shows the structure of one Embodiment of an information processing system. データベースの構成を示す図である。It is a figure which shows the structure of a database. 情報処理システムの動作について説明するためのフローチャートである。It is a flowchart for demonstrating operation of an information processing system. ２台の情報処理装置で連動した動作を行う場合について説明するための図である。It is a figure for demonstrating the case where two information processing apparatus perform interlocking operation. サーバの構成例を示す図である。It is a figure which shows the configuration example of a server. 記録媒体について説明するための図である。It is a figure for demonstrating the recording medium.
　以下に、本技術を実施するための形態（以下、実施の形態という）について説明する。 The embodiment for implementing the present technology (hereinafter referred to as the embodiment) will be described below.
　本技術は、図１に示すように、情報処理装置１０としてのスマートフォンで撮像した画像に対応する振動を、情報処理装置１０で提示する場合に適用できる。以下の説明では、情報処理装置１０としてスマートフォンを例に挙げて説明するが、他の装置であっても、以下に説明する本技術を適用することはできる。 As shown in FIG. 1, this technique can be applied when the information processing device 10 presents vibrations corresponding to an image captured by a smartphone as the information processing device 10. In the following description, a smartphone will be taken as an example as the information processing device 10, but the present technology described below can be applied to other devices as well.
　＜情報処理装置のハードウエア構成例＞
　図２は、本技術を適用した情報処理装置１０としてのスマートフォンの一実施の形態のハードウエア構成例を示すブロック図である。 <Hardware configuration example of information processing device>
FIG. 2 is a block diagram showing a hardware configuration example of an embodiment of a smartphone as an information processing device 10 to which the present technology is applied.
　図１において、情報処理装置１０は、ＣＰＵ（Central Processing Unit)１１，ＲＯＭ（Read Only Memory)１２，ＲＡＭ（Random Access Memory)１３、センサ部１４、入出力インタフェース１５、表示部１６、タッチ入力部１７、通信部１８、および、ホームボタン１９等を有する。 In FIG. 1, the information processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a sensor unit 14, an input / output interface 15, a display unit 16, and a touch input unit. It has 17, a communication unit 18, a home button 19, and the like.
　ＣＰＵ１１乃至入出力インタフェース１５は、バスを介して相互に接続されている。表示部１６乃至ホームボタン１９は、入出力インタフェース１５に接続されている。 The CPU 11 to the input / output interface 15 are connected to each other via a bus. The display unit 16 to the home button 19 are connected to the input / output interface 15.
　ＣＰＵ１１は、ＲＯＭ１２やＲＡＭ１３に記憶されたプログラムを実行することにより、各種の処理を行う。ＲＯＭ１２およびＲＡＭ１３は、ＣＰＵ１１が実行するプログラムや、ＣＰＵ１１の動作上必要なデータを記憶する。 The CPU 11 performs various processes by executing a program stored in the ROM 12 or the RAM 13. The ROM 12 and the RAM 13 store a program executed by the CPU 11 and data necessary for the operation of the CPU 11.
　センサ部１４は、加速度センサや、各種の光センサ、その他の必要なセンサを含み、所定の物理量をセンシングする。光センサとしては、例えば、RGB画像を撮影するモノラルRGBカメラや、白黒（グレスケール）の画像を撮影するモノラルグレイカメラ、距離を画素値とする距離画像を得る測距センサ等がある。測距センサとしては、TOFセンサや、ストラクチャドライトカメラ、（パターンド）ステレオカメラ等がある。光センサは、表示部１６側や、表示部１６の裏面側に設けることができる。 The sensor unit 14 includes an acceleration sensor, various optical sensors, and other necessary sensors, and senses a predetermined physical quantity. Examples of the optical sensor include a monaural RGB camera that captures an RGB image, a monaural gray camera that captures a black-and-white (grescale) image, and a ranging sensor that obtains a distance image whose pixel value is a distance. Distance measurement sensors include TOF sensors, structured light cameras, (patterned) stereo cameras, and the like. The optical sensor can be provided on the display unit 16 side or the back surface side of the display unit 16.
　入出力インタフェース１５は、表示部１６乃至ホームボタン１９その他のデバイスのインタフェースとして機能する。表示部１６は、画像を表示する。タッチ入力部１７は、ユーザのタッチ操作（タップやフリップ等）を受け付ける。表示部１６およびタッチ入力部１７は、タッチパネル等によって一体的に構成することができる。 The input / output interface 15 functions as an interface for the display unit 16 to the home button 19 and other devices. The display unit 16 displays an image. The touch input unit 17 accepts a user's touch operation (tap, flip, etc.). The display unit 16 and the touch input unit 17 can be integrally configured by a touch panel or the like.
　通信部１８は、無線LAN（Local Area Network）等の所定の通信方式の通信を行う。ホームボタン１９は、情報処理装置１０の下部に設けられた物理的なまたは仮想的なボタンである。振動子２０は、情報処理装置１０に振動を与える機能を有する。 The communication unit 18 communicates by a predetermined communication method such as a wireless LAN (Local Area Network). The home button 19 is a physical or virtual button provided at the bottom of the information processing device 10. The oscillator 20 has a function of giving vibration to the information processing device 10.
　以上のように構成される情報処理装置１０では、ＣＰＵ１１（コンピュータ）が、ＲＯＭ１２やＲＡＭ１３に予めインストールされたプログラムや、通信部１８によってダウンロードされ、ＲＯＭ１２やＲＡＭ１３にインストールされたプログラムを実行することにより、各種の処理を行う。これにより、情報処理装置１０は、以下に説明する撮像された物体に応じた振動を、ユーザに提示する。 In the information processing device 10 configured as described above, the CPU 11 (computer) executes a program pre-installed in the ROM 12 or the RAM 13 or a program downloaded by the communication unit 18 and installed in the ROM 12 or the RAM 13. , Perform various processes. As a result, the information processing apparatus 10 presents the user with vibrations corresponding to the imaged object described below.
　なお、コンピュータとしてのＣＰＵ１１がプログラムに従って行う処理は、必ずしも、後述するフローチャートとして記載された順序に沿って時系列に行われる必要はない。すなわち、コンピュータがプログラムに従って行う処理は、並列的あるいは個別に実行される処理（例えば、並列処理あるいはオブジェクトによる処理）も含む。 Note that the processing performed by the CPU 11 as a computer according to the program does not necessarily have to be performed in chronological order in the order described as the flowchart described later. That is, the processing performed by the computer according to the program also includes processing executed in parallel or individually (for example, parallel processing or processing by an object).
　また、プログラムは、１のコンピュータ（プロセッサ）により処理されるものであっても良いし、複数のコンピュータによって分散処理されるものであっても良い。さらに、プログラムは、遠方のコンピュータに転送されて実行されるものであっても良い。 Further, the program may be processed by one computer (processor) or may be distributed by a plurality of computers. Further, the program may be transferred to a distant computer and executed.
　＜情報処理装置の機能＞
　図３は、情報処理装置１０の機能構成例を示す図である。情報処理装置１０は、撮像部４１、物体認識部４２、振動データ取得部４３、振動データ記憶部４４、差分判定部４５、属性パラメータ取得部２２５、属性パラメータ記憶部４７、補正処理部４８、および振動生成提示部４９を備える。 <Function of information processing device>
FIG. 3 is a diagram showing a functional configuration example of the information processing device 10. The information processing device 10 includes an imaging unit 41, an object recognition unit 42, a vibration data acquisition unit 43, a vibration data storage unit 44, a difference determination unit 45, an attribute parameter acquisition unit 225, an attribute parameter storage unit 47, a correction processing unit 48, and A vibration generation presentation unit 49 is provided.
　撮像部４１は、イメージセンサを含むカメラであり、物体を撮像する。撮像部４１で撮像された物体画像のデータは、物体認識部４２に供給される。物体認識部４２は、撮像された画像内の物体を認識する。認識された物体の情報（以下、適宜、認識物体情報と記述する）は、振動データ取得部４３と差分判定部４５に供給される。物体認識情報は、例えば、所定の物体、その物体の個数や量などの情報である。 The imaging unit 41 is a camera including an image sensor and images an object. The data of the object image captured by the image pickup unit 41 is supplied to the object recognition unit 42. The object recognition unit 42 recognizes an object in the captured image. The recognized object information (hereinafter, appropriately referred to as recognized object information) is supplied to the vibration data acquisition unit 43 and the difference determination unit 45. The object recognition information is, for example, information such as a predetermined object and the number and amount of the objects.
　振動データ取得部４３は、振動データ記憶部４４に記憶されている振動データであり、認識物体情報に基づく振動データを取得する。振動データ記憶部４４には、物体と、その物体に関する振動データが記憶されている。また、その振動データは、物体の個数や量などと関連付けられて記憶されている。 The vibration data acquisition unit 43 is vibration data stored in the vibration data storage unit 44, and acquires vibration data based on the recognized object information. The vibration data storage unit 44 stores an object and vibration data related to the object. In addition, the vibration data is stored in association with the number and quantity of objects.
　例えば認識物体情報が、認識された物体として馬である場合、振動データ取得部４３は、馬が疾走しているときの振動を表す振動データを、振動データ記憶部４４から読み出す。また、その読み出される振動データは、例えば、馬１頭という物体の個数と関連付けられて記憶されていたデータである。 For example, when the recognized object information is a horse as the recognized object, the vibration data acquisition unit 43 reads the vibration data representing the vibration when the horse is running from the vibration data storage unit 44. Further, the vibration data read out is, for example, data stored in association with the number of objects such as one horse.
　振動データ取得部４３により取得された振動データは、差分判定部４５に供給される。差分判定部４５には、物体認識部４２からの物体認識情報も供給される。差分判定部４５は、物体認識情報が表す物体の個数や量（以下、個数を例に挙げて説明する）と、振動データに関連付けられている物体の個数の差分を判定する。差分判定部４５は、物体認識情報が表す物体の状態と、振動データに関連付けられている物体の状態を比較し、差分を算出する。 The vibration data acquired by the vibration data acquisition unit 43 is supplied to the difference determination unit 45. The object recognition information from the object recognition unit 42 is also supplied to the difference determination unit 45. The difference determination unit 45 determines the difference between the number and amount of objects represented by the object recognition information (hereinafter, the number will be described as an example) and the number of objects associated with the vibration data. The difference determination unit 45 compares the state of the object represented by the object recognition information with the state of the object associated with the vibration data, and calculates the difference.
　例えば、物体認識情報が表す物体が馬であり、その物体の個数が２頭であり、振動データに関連付けられている個数が１頭である場合、１頭分の差分があると判定される。 For example, if the object represented by the object recognition information is a horse, the number of the objects is two, and the number associated with the vibration data is one, it is determined that there is a difference for one horse.
　振動データに関連付けられている物体の個数の情報は、画像であっても良い。振動データに関連付けられている物体が撮像された画像であり、その画像と、物体認識情報としての撮像された画像が比較されることで、物体の差分が抽出されるようにしても良い。 The information on the number of objects associated with the vibration data may be an image. The object associated with the vibration data is an image captured, and the difference between the objects may be extracted by comparing the image with the captured image as the object recognition information.
　差分判定部４５は、振動データ取得部４３から供給された振動データを、補正処理部４８に供給（転送）する。また、差分判定部４５は、差分があると判定した場合、属性パラメータ取得部２２５に属性パラメータを取得するように指示を出す。属性パラメータ取得部２２５は、属性パラメータ記憶部４７から属性パラメータを取得する。 The difference determination unit 45 supplies (transfers) the vibration data supplied from the vibration data acquisition unit 43 to the correction processing unit 48. Further, when the difference determination unit 45 determines that there is a difference, the difference determination unit 45 instructs the attribute parameter acquisition unit 225 to acquire the attribute parameter. The attribute parameter acquisition unit 225 acquires the attribute parameter from the attribute parameter storage unit 47.
　振動データ記憶部４４に記憶されている振動データは、物体が所定の個数である場合の振動に関するデータである。以下、振動データ記憶部４４に記憶されている振動データを、適宜、基準振動データと記述する。また基準振動データと関連付けられている物体の個数や量を、基準個数（量）と記述する。 The vibration data stored in the vibration data storage unit 44 is data related to vibration when the number of objects is a predetermined number. Hereinafter, the vibration data stored in the vibration data storage unit 44 will be appropriately referred to as reference vibration data. The number and amount of objects associated with the reference vibration data are described as the reference number (quantity).
　物体認識情報が表す物体の個数が、基準個数と異なる場合、基準振動データを、物体認識情報が表す物体の個数に合う振動データに変換するためのパラメータが、属性パラメータである。 When the number of objects represented by the object recognition information is different from the reference number, the parameter for converting the reference vibration data into the vibration data matching the number of objects represented by the object recognition information is the attribute parameter.
　補正処理部４８は、差分判定部４５から、差分がないとの情報が供給された場合、差分判定部４５を介して供給された振動データを、振動生成提示部４９に供給する。補正処理部４８は、差分判定部４５から、差分があるとの情報が供給された場合、差分判定部４５を介して供給された振動データを、属性パラメータ取得部２２５から供給される属性パラメータを用いて補正し、振動生成提示部４９に供給する。 When the difference determination unit 45 supplies information that there is no difference, the correction processing unit 48 supplies the vibration data supplied via the difference determination unit 45 to the vibration generation presentation unit 49. When the difference determination unit 45 supplies information that there is a difference, the correction processing unit 48 uses the vibration data supplied through the difference determination unit 45 as the attribute parameter supplied from the attribute parameter acquisition unit 225. It is corrected by using and supplied to the vibration generation presentation unit 49.
　振動生成提示部４９は、供給された振動データに基づく振動を生成し、ユーザに提示する。 The vibration generation presentation unit 49 generates vibration based on the supplied vibration data and presents it to the user.
　＜データベースの構成例＞
　図４に振動データ記憶部４４と属性パラメータ記憶部４７のデータベース構成例を示す。なお、図３では、説明のため振動データ記憶部４４と属性パラメータ記憶部４７を別々に図示したが、図４に示すように１つのデータベースとすることができ、図４では、１つのデータベースとした場合の構成例を示す。 <Database configuration example>
FIG. 4 shows a database configuration example of the vibration data storage unit 44 and the attribute parameter storage unit 47. Although the vibration data storage unit 44 and the attribute parameter storage unit 47 are shown separately in FIG. 3 for explanation, they can be combined into one database as shown in FIG. 4, and in FIG. 4, one database is used. An example of the configuration is shown below.
　振動データ記憶部４４は、物体名、アクション、振動データ、および属性データが関連付けられて記憶されている。また、属性パラメータ記憶部４７は、少なくとも物体名と属性パラメータが関連付けられて記憶されている。 The vibration data storage unit 44 stores the object name, the action, the vibration data, and the attribute data in association with each other. Further, the attribute parameter storage unit 47 stores at least the object name and the attribute parameter in association with each other.
　物体名の欄には、物体名が記載される。物体名は、用意されている振動データを識別する識別子としての情報である。ここでは物体名としたが、物体の画像、音声などでも良いし、所定の物体に割り当てられるＩＤなどでも良い。 The object name is entered in the object name column. The object name is information as an identifier that identifies the prepared vibration data. Although the object name is used here, it may be an image, sound, or the like of the object, or an ID assigned to a predetermined object.
　アクションの欄には、振動を提示するときに、ユーザからの何らかのアクションが必要な場合、そのアクションが記載される。振動データの欄には、提示する振動データ（基準振動データ）の情報が記憶される。振動データの情報としては、振動の大きさ、ピッチ、時間、余韻などがある。 In the action column, if any action from the user is required when presenting vibration, that action is described. Information on the presented vibration data (reference vibration data) is stored in the vibration data column. The vibration data information includes vibration magnitude, pitch, time, and afterglow.
　属性データの欄には、物体の個数や量に関するデータが記載される。属性データの欄には、基準となる基準画像が記載されていても良い。基準画像とは、所定の物体の基準となる状態が撮像されている画像である。例えば、所定の容器に水が満タンな状態で充填されている画像であり、このような画像は物体と、物体の量が関連付けられた画像として扱うことができる。 In the attribute data column, data related to the number and quantity of objects is described. A reference image as a reference may be described in the attribute data column. The reference image is an image in which a reference state of a predetermined object is captured. For example, it is an image in which a predetermined container is filled with water, and such an image can be treated as an image in which an object and an amount of the object are associated with each other.
　属性パラメータ記憶部４７の属性パラメータの欄には、属性データに記載されている物体の個数や量以外の個数や量以外のときに、振動パラメータを補正するための情報が記載されている。 In the attribute parameter column of the attribute parameter storage unit 47, information for correcting the vibration parameter is described when the number or quantity is other than the number or quantity of the objects described in the attribute data.
　振動データの欄に記載されている振動データを、基準振動データとした場合、基準振動データは、属性データの欄に記載されている物体の個数のときの振動を提示するためのデータである。属性データの欄に記載されている物体の個数以外の個数のときの振動データを生成するための情報が、属性パラメータである。 When the vibration data described in the vibration data column is used as the reference vibration data, the reference vibration data is data for presenting the vibration when the number of objects is the number of objects described in the attribute data column. The information for generating the vibration data when the number of objects other than the number of objects described in the attribute data column is other than the number of objects is the attribute parameter.
　例えば、物体名が馬（静止）、アクションが空欄、振動データが馬振動データＡ、属性データが１頭という情報が関連付けられている。これらのデータは、図５のＡに示したように、静止している馬１０１のときの基準振動データが、馬振動データＡであることを意味している。また、馬振動データＡは、ユーザのアクションによらず、例えば、静止している馬の画像が撮像された時点で振動の提示が開始される。 For example, information that the object name is horse (stationary), the action is blank, the vibration data is horse vibration data A, and the attribute data is one horse is associated. These data mean that, as shown in A of FIG. 5, the reference vibration data when the horse 101 is stationary is the horse vibration data A. Further, in the horse vibration data A, the vibration presentation is started when, for example, an image of a stationary horse is captured, regardless of the user's action.
　また属性パラメータとして、頭数倍という情報が記載されている。この場合、基準振動データを頭数倍することで、頭数分の振動データが生成される。例えば、図５のＢに示したように、情報処理装置１０により、２頭の静止している馬１０２と馬１０３が撮像され、表示部１６に表示された場合、馬振動データＡに、２を乗算することで、２頭の馬がいるときの振動データが生成され、情報処理装置１０によりユーザに提示される。 Also, as an attribute parameter, information that the number of heads is multiplied is described. In this case, by multiplying the reference vibration data by the number of heads, vibration data for the number of heads is generated. For example, as shown in FIG. 5B, when two stationary horses 102 and 103 are imaged by the information processing device 10 and displayed on the display unit 16, the horse vibration data A contains 2 By multiplying, vibration data when there are two horses is generated and presented to the user by the information processing device 10.
　この場合、差分判定部４５は、認識物体情報から、静止している馬が２頭いるという状態と、属性データから、静止している馬が１頭いるという状態を比較し、その比較結果を、差分判定結果とする。この差分判定結果に応じて、上記したような補正（生成）の処理が実行される。 In this case, the difference determination unit 45 compares the state where there are two stationary horses from the recognized object information and the state where there is one stationary horse from the attribute data, and compares the comparison result. , The difference judgment result. The correction (generation) process as described above is executed according to the difference determination result.
　属性データとして、静止している１頭の馬の画像を基準画像、例えば、図５のＡに示した画像を基準画像として記憶しておき、この基準画像との差異が抽出されるようにしても良い。撮像部４１で撮像された画像が、静止している２頭の馬の画像であった場合、この撮像画像と、基準画像が比較され、基準画像との差分、この場合、馬１頭の違いがあるとの差分が抽出されるようにしても良い。また、この抽出された差分に応じた振動データに補正されるようにしても良い。 As attribute data, an image of one stationary horse is stored as a reference image, for example, the image shown in FIG. 5A is stored as a reference image so that the difference from this reference image can be extracted. Is also good. When the image captured by the imaging unit 41 is an image of two horses that are stationary, the captured image and the reference image are compared, and the difference from the reference image, in this case, the difference of one horse. The difference between the existence and the presence may be extracted. Further, the vibration data may be corrected according to the extracted difference.
　図４を参照するに例えば、物体名が馬（疾走）、アクションが空欄、振動データが馬振動データＢ、属性データが１頭という情報が関連付けられている。これらのデータは、図６のＡに示したように、疾走している馬１１１のときの基準振動データが、馬振動データＢであることを意味している。また、馬振動データＢは、ユーザのアクションによらず、例えば、疾走している馬の画像が撮像された時点で振動の提示が開始される。 With reference to FIG. 4, for example, information that the object name is horse (sprinting), the action is blank, the vibration data is horse vibration data B, and the attribute data is one horse is associated. These data mean that, as shown in A of FIG. 6, the reference vibration data at the time of the running horse 111 is the horse vibration data B. Further, in the horse vibration data B, the vibration presentation is started when, for example, an image of a running horse is captured, regardless of the user's action.
　また属性パラメータとして、頭数倍という情報が記載されている。この場合、基準振動データを頭数倍することで、頭数分の振動データが生成される。例えば、図６のＢに示したように、情報処理装置１０により、２頭の疾走している馬１１２と馬１１３が撮像され、表示部１６に表示された場合、馬振動データＢに、２を乗算することで、２頭の馬が疾走しているときの振動データが生成され、情報処理装置１０によりユーザに提示される。 Also, as an attribute parameter, information that the number of heads is multiplied is described. In this case, by multiplying the reference vibration data by the number of heads, vibration data for the number of heads is generated. For example, as shown in B of FIG. 6, when two running horses 112 and 113 are imaged by the information processing device 10 and displayed on the display unit 16, the horse vibration data B shows 2 By multiplying by, vibration data when two horses are running is generated and presented to the user by the information processing device 10.
　この場合、差分判定部４５は、認識物体情報から、疾走している馬が２頭いるという状態と、属性データから、疾走している馬が１頭いるという状態を比較し、その比較結果を、差分判定結果とする。この差分判定結果に応じて、上記したような補正（生成）の処理が実行される。 In this case, the difference determination unit 45 compares the state where there are two running horses from the recognized object information and the state where there is one running horse from the attribute data, and compares the comparison result. , The difference judgment result. The correction (generation) process as described above is executed according to the difference determination result.
　属性データとして、疾走している１頭の馬の画像を基準画像、例えば、図６のＡに示した画像を基準画像として記憶しておき、この基準画像との差異が抽出されるようにしても良い。撮像部４１で撮像された画像が、疾走している２頭の馬の画像であった場合、この撮像画像と、基準画像が比較され、基準画像との差分、この場合、馬１頭の違いがあるとの差分が抽出されるようにしても良い。また、この抽出された差分に応じた振動データに補正されるようにしても良い。 As attribute data, an image of one running horse is stored as a reference image, for example, the image shown in FIG. 6A is stored as a reference image so that the difference from this reference image can be extracted. Is also good. When the image captured by the imaging unit 41 is an image of two horses running, the captured image and the reference image are compared, and the difference from the reference image, in this case, the difference of one horse. The difference between the existence and the presence may be extracted. Further, the vibration data may be corrected according to the extracted difference.
　図４を参照するに例えば、物体名が水（振る）、アクションが上下振動、振動データが水振動データＡ、属性データが５００ｍｌという情報が関連付けられている。これらのデータは、図７のＡに示したように、ペットボトルなどの容器１３１に５００ｍｌ（満水の状態）のときに、容器１３１が上下に振られたときの基準振動データが、水振動データＡであることを意味している。 With reference to FIG. 4, for example, information that the object name is water (shaking), the action is vertical vibration, the vibration data is water vibration data A, and the attribute data is 500 ml is associated. As shown in A of FIG. 7, these data are based on the water vibration data when the container 131 is shaken up and down when the container 131 such as a PET bottle is 500 ml (full state). It means that it is A.
　また、水振動データＡは、ユーザが、アクションとして上下振動、例えば情報処理装置１０を上下に揺らしたり、情報処理装置１０の画面を指でタッチしながら上下に動かしたりするなどのアクションがされたときに、水が容器１３１内で上下に振動しているときの振動が、ユーザに提示される。 Further, in the water vibration data A, the user takes an action such as vertical vibration, for example, shaking the information processing device 10 up and down, or moving the screen of the information processing device 10 up and down while touching it with a finger. Occasionally, the vibration of water vibrating up and down in the container 131 is presented to the user.
　また属性パラメータとして、（水量×ａ）倍という情報が記載されている。この場合、基準振動データを（水量×ａ）倍することで、水量に合った振動データが生成される。例えば、図７のＢに示したように、情報処理装置１０により、水が２００ｍｌ入っている容器１３２が撮像される。そして、図８のＡに示したように、ユーザが表示部１６（タッチパネル）上を、指でタッチしながら上下に動かすと、振動の提示が開始される。この場合、２００ｍｌの水が、容器１３２内で上下に動くときの振動の振動データが、水振動データＡに、（２００×ａ）が乗算されることで生成され、その振動データに基づく振動が、ユーザに提示される。 Also, as an attribute parameter, information of (water volume x a) times is described. In this case, by multiplying the reference vibration data (water amount × a), vibration data matching the water amount is generated. For example, as shown in B of FIG. 7, the information processing apparatus 10 images a container 132 containing 200 ml of water. Then, as shown in FIG. 8A, when the user moves the display unit 16 (touch panel) up and down while touching it with a finger, the presentation of vibration is started. In this case, vibration data of vibration when 200 ml of water moves up and down in the container 132 is generated by multiplying water vibration data A by (200 × a), and vibration based on the vibration data is generated. , Presented to the user.
　この場合、差分判定部４５は、認識物体情報から、２００ｍｌの水があるという状態と、属性データから、５００ｍｌの水があるという状態を比較し、その比較結果を、差分判定結果とする。この差分判定結果に応じて、上記したような補正（生成）の処理が実行される。 In this case, the difference determination unit 45 compares the state where there is 200 ml of water from the recognized object information and the state where there is 500 ml of water from the attribute data, and sets the comparison result as the difference determination result. The correction (generation) process as described above is executed according to the difference determination result.
　属性データとして、５００ｍｌの水が入っている容器の画像を基準画像、例えば、図７のＡに示したような画像を基準画像として記憶しておき、この基準画像との差異が抽出されるようにしても良い。撮像部４１で撮像された画像が、２００ｍｌの水が入っている容器の画像であった場合、この撮像画像と、基準画像が比較され、基準画像との差分、この場合、３００ｍｌ少ないとの違いが差分として抽出されるようにしても良い。また、この抽出された差分に応じた振動データに補正されるようにしても良い。 As attribute data, an image of a container containing 500 ml of water is stored as a reference image, for example, an image as shown in A of FIG. 7 is stored as a reference image, and a difference from this reference image is extracted. You can do it. When the image captured by the imaging unit 41 is an image of a container containing 200 ml of water, the captured image is compared with the reference image, and the difference from the reference image, in this case, the difference of 300 ml less. May be extracted as a difference. Further, the vibration data may be corrected according to the extracted difference.
　図４を参照するに例えば、物体名が水（注ぐ）、アクションが傾ける、振動データが水振動データＢ、属性データが５００ｍｌという情報が関連付けられている。これらのデータは、図７のＡに示したように、ペットボトルなどの容器１３１に５００ｍｌ（満水の状態）のときに、容器１３１が傾けられたとき（水が容器１３１からコップなどに注がれるとき）の基準振動データが、水振動データＢであることを意味している。 With reference to FIG. 4, for example, information that the object name is water (pouring), the action is tilted, the vibration data is water vibration data B, and the attribute data is 500 ml is associated. As shown in A of FIG. 7, these data are obtained when the container 131 such as a PET bottle is 500 ml (full state) and the container 131 is tilted (water is poured from the container 131 into a cup or the like). It means that the reference vibration data of) is the water vibration data B.
　また、水振動データＢは、ユーザが、アクションとして傾ける、例えば情報処理装置１０を傾けたり、情報処理装置１０の画面を指でタッチしながら容器１３１の口から水が出るような方向に動かしたりするなどのアクションがされたときに、水が容器１３１からコップなどに注がれるときの振動が、ユーザに提示される。 Further, the water vibration data B is tilted as an action by the user, for example, tilting the information processing device 10 or moving the water vibration data B in a direction in which water comes out from the mouth of the container 131 while touching the screen of the information processing device 10 with a finger. When an action such as is performed, the vibration when water is poured from the container 131 into a cup or the like is presented to the user.
　また属性パラメータとして、（水量×ｂ）倍という情報が記載されている。この場合、基準振動データを（水量×ｂ）倍することで、水量に合った振動データが生成される。例えば、図７のＢに示したように、情報処理装置１０により、水が２００ｍｌ入っている容器１３２が撮像される。そして、図８のＢに示したように、ユーザが情報処理装置１０を傾けると、振動の提示が開始される。この場合、２００ｍｌの水が、容器１３２から注がれるときの振動の振動データが、水振動データＢに、（２００×ｂ）が乗算されることで生成され、その振動データに基づく振動が、ユーザに提示される。 Also, as an attribute parameter, information of (water volume x b) times is described. In this case, by multiplying the reference vibration data (water amount × b), vibration data matching the water amount is generated. For example, as shown in B of FIG. 7, the information processing apparatus 10 images a container 132 containing 200 ml of water. Then, as shown in FIG. 8B, when the user tilts the information processing device 10, the presentation of vibration is started. In this case, the vibration data of the vibration when 200 ml of water is poured from the container 132 is generated by multiplying the water vibration data B by (200 × b), and the vibration based on the vibration data is generated. Presented to the user.
　図４を参照するに例えば、物体名が車（アイドリング）、アクションがタップ、振動データが車振動データＡ、属性データが１台という情報が関連付けられている。これらのデータは、図９のＡに示したように、アイドリングしている車１５１のときの基準振動データが、車振動データＡであることを意味している。また、車振動データＡは、ユーザが情報処理装置１０をタップしたときに振動の提示が開始される。 With reference to FIG. 4, for example, information that the object name is a car (idling), the action is a tap, the vibration data is the vehicle vibration data A, and the attribute data is one is associated. These data mean that, as shown in A of FIG. 9, the reference vibration data when the idling vehicle 151 is the vehicle vibration data A. Further, the vehicle vibration data A starts to be presented with vibration when the user taps the information processing device 10.
　この場合、差分判定部４５は、認識物体情報から、アイドリングしている２台の車があるという状態と、属性データから、アイドリングしている１台の車があるという状態を比較し、その比較結果を、差分判定結果とする。この差分判定結果に応じて、上記したような補正（生成）の処理が実行される。 In this case, the difference determination unit 45 compares the state where there are two idling cars from the recognized object information and the state where there is one idling car from the attribute data, and compares them. The result is used as the difference determination result. The correction (generation) process as described above is executed according to the difference determination result.
　属性データとして、アイドリングしている１台の車の画像を基準画像、例えば、図９のＡに示した画像を基準画像として記憶しておき、この基準画像との差異が抽出されるようにしても良い。撮像部４１で撮像された画像が、アイドリングしている２台の車の画像であった場合、この撮像画像と、基準画像が比較され、基準画像との差分、この場合、車１台の違いがあるとの差分が抽出されるようにしても良い。また、この抽出された差分に応じた振動データに補正されるようにしても良い。 As attribute data, an image of one idling car is stored as a reference image, for example, the image shown in FIG. 9A is stored as a reference image so that the difference from this reference image can be extracted. Is also good. When the image captured by the imaging unit 41 is an image of two idling cars, the captured image is compared with the reference image, and the difference from the reference image, in this case, the difference of one car. The difference between the existence and the presence may be extracted. Further, the vibration data may be corrected according to the extracted difference.
　また属性パラメータとして、台数倍という情報が記載されている。この場合、基準振動データを台数倍することで、台数分の振動データが生成される。例えば、図９のＢに示したように、情報処理装置１０により、２台のアイドリングしている車１５２と車１５３が撮像され、表示部１６に表示された場合、車振動データＡに、２を乗算することで、２台の車がいるときの振動データが生成され、情報処理装置１０によりユーザに提示される。 Also, as an attribute parameter, information that the number of units is doubled is described. In this case, by multiplying the reference vibration data by the number of units, vibration data for the number of units is generated. For example, as shown in B of FIG. 9, when two idling vehicles 152 and 153 are imaged by the information processing device 10 and displayed on the display unit 16, the vehicle vibration data A contains 2 By multiplying, vibration data when there are two cars is generated and presented to the user by the information processing device 10.
　このように、データベースとして、基準振動データや、その基準振動データを補正するためのパラメータなどが、情報処理装置１０には記憶されている。 In this way, as a database, the reference vibration data and the parameters for correcting the reference vibration data are stored in the information processing apparatus 10.
　なお、図４乃至図９を参照して説明したデータや動作（アクション）は、一例であり、限定を示す記載ではない。例えば、アクションとしては、物体に近づく（撮像されている物体が大きくなる、位置データにより、物体に近づいていると判定できるなど）、なぞるなどであっても良いし、ユーザからの音声により指示があったときなどでも良い。また、アクションとして、タッチパネルを押した力なども考慮されても良い。 Note that the data and actions described with reference to FIGS. 4 to 9 are examples and do not indicate limitations. For example, the action may be approaching an object (the imaged object becomes larger, it can be determined from the position data that the object is approaching, etc.), tracing, etc., or an instruction is given by voice from the user. It may be when there is. Further, as an action, the force of pressing the touch panel may be taken into consideration.
　またアクションの種類により、異なる振動が提示されるようにしても良い。例えば、図４に示した物体名として車（アイドリング）のアクションとして、タップの場合には、上記した処理が行われ、なぞる（スライド）のときには、車が走行しているときの振動が提示される属性パラメータが選択された処理が行われるようにしても良い。 Also, different vibrations may be presented depending on the type of action. For example, as an action of a car (idling) as an object name shown in FIG. 4, the above processing is performed in the case of tapping, and vibration when the car is running is presented in the case of tracing (sliding). The processing in which the attribute parameter is selected may be performed.
　このように、情報処理装置１０（のデータベース）には、基準振動データや、その基準音声データを補正するためのパラメータなどが記憶されている。また、基準振動データに関する情報として、物体の個数や量を含む物体の状態に関する情報を記憶しておき、その情報との差分が抽出され、その差分に応じた振動データに変換される。よって、ユーザに、物体の状態に応じた振動を提示することができる。 In this way, the information processing device 10 (database) stores the reference vibration data and parameters for correcting the reference voice data. Further, as information on the reference vibration data, information on the state of the object including the number and quantity of the objects is stored, the difference from the information is extracted, and the information is converted into vibration data according to the difference. Therefore, it is possible to present the user with vibration according to the state of the object.
　また、基準振動データに関する情報として、物体の個数や量を含む物体の状態に関する画像を基準画像として用意しておき、その基準画像と撮像された画像との差分が抽出され、その差分に応じた振動データに変換される。よって、ユーザに、物体の状態に応じた振動を提示することができる。 Further, as information on the reference vibration data, an image relating to the state of the object including the number and quantity of the objects is prepared as a reference image, and the difference between the reference image and the captured image is extracted and corresponding to the difference. Converted to vibration data. Therefore, it is possible to present the user with vibration according to the state of the object.
　＜情報処理装置の動作＞
　図１０のフローチャートを参照し、上記したような振動を提示する情報処理装置１０の動作について説明する。 <Operation of information processing device>
The operation of the information processing apparatus 10 that presents the vibration as described above will be described with reference to the flowchart of FIG.
　ステップＳ１１において、撮像部４１により画像が撮像される。図１に示したように、ユーザが、情報処理装置１０を保持している状態で、所定の物体にカメラを向け、撮像が行われる。 In step S11, the image is captured by the imaging unit 41. As shown in FIG. 1, the user points the camera at a predetermined object while holding the information processing device 10, and imaging is performed.
　ステップＳ１２において、物体認識部４２は、撮像部４１で撮像された画像を解析し、画像内の物体、その物体の個数や量、状態などを検出し、認識物体情報として、振動データ取得部４３と差分判定部４５に供給する。 In step S12, the object recognition unit 42 analyzes the image captured by the image pickup unit 41, detects the object in the image, the number, amount, state, etc. of the object, and uses the vibration data acquisition unit 43 as the recognition object information. Is supplied to the difference determination unit 45.
　ステップＳ１３において、振動データ取得部４３は、振動データ記憶部４４から、認識物体情報が示す物体（物体名）に関連付けられているデータを読み出す。読み出されるデータは、図４を参照して説明したように、アクション、振動データ、属性データである。読み出されたデータは、差分判定部４５に供給される。 In step S13, the vibration data acquisition unit 43 reads the data associated with the object (object name) indicated by the recognized object information from the vibration data storage unit 44. The data to be read is action, vibration data, and attribute data as described with reference to FIG. The read data is supplied to the difference determination unit 45.
　ステップＳ１４において、差分判定部４５は、物体情報についての差分があるか否かを判定する。差分判定部４５は、振動データ取得部４３から供給された属性データと、物体認識部４２から供給された認識物体情報を参照し、属性データで示される個数（量）と、認識物体情報で示される個数（量）を比較し、差分があるか否かを判定する。 In step S14, the difference determination unit 45 determines whether or not there is a difference in the object information. The difference determination unit 45 refers to the attribute data supplied from the vibration data acquisition unit 43 and the recognized object information supplied from the object recognition unit 42, and indicates the number (quantity) indicated by the attribute data and the recognized object information. Compare the number (quantity) to be obtained and determine whether or not there is a difference.
　例えば、図５を参照して説明したように、２頭の馬が撮影された場合、認識物体情報で示される個数が２であり、属性データが示す個数が１であるため、差分があるとの判定が出される。また、１頭の馬が撮影された場合、認識物体情報で示される個数が１であり、属性データが示す個数が１であるため、差分はないとの判定が出される。 For example, as described with reference to FIG. 5, when two horses are photographed, the number indicated by the perceptible object information is 2, and the number indicated by the attribute data is 1, so that there is a difference. Is issued. Further, when one horse is photographed, the number indicated by the recognized object information is 1, and the number indicated by the attribute data is 1, so it is determined that there is no difference.
　また例えば、図７、図８を参照して説明したように、５００ｍｌ容器に２００ｍｌ程度の水が入っている容器が撮影された場合、認識物体情報で示される量が２００であり、属性データが示す量が５００であるため、差分があるとの判定が出される。また、５００ｍｌの水が入っている容器が撮影された場合、認識物体情報で示される量が５００であり、属性データが示す量が５００であるため、差分はないとの判定が出される。 Further, for example, as described with reference to FIGS. 7 and 8, when a container containing about 200 ml of water is photographed in a 500 ml container, the amount indicated by the perceptible object information is 200, and the attribute data is Since the indicated amount is 500, it is determined that there is a difference. Further, when a container containing 500 ml of water is photographed, it is determined that there is no difference because the amount indicated by the recognized object information is 500 and the amount indicated by the attribute data is 500.
　なおここでは、水の量として５００ｍｌや２００ｍｌといった具体的な数値を挙げたが、正確な量が検出されなくても良く、属性データとしては満タンな状態（未開封な状態）とし、検出される水の量としては、２／３程度、１／２程度といった概算でも良い。 Here, specific numerical values such as 500 ml and 200 ml are given as the amount of water, but the exact amount does not have to be detected, and the attribute data is detected as a full state (unopened state). The amount of water to be collected may be estimated to be about 2/3 or 1/2.
　また、水のような量を比較する場合、画像同士を比較することで、差分が算出されるようにしても良い。例えば、属性データとして基準画像を記憶し、その基準画像との差分が抽出されるようにしても良い。例えば、基準画像としては、５００ｍｌの水が入っている容器の画像が用意され、その基準画像と撮影された２００ｍｌの水が入っている容器の画像が比較され、基準画像との差分、この場合、３００ｍｌの量が違いあるという差分が抽出されるようにしても良い。そして、この抽出された差分に応じた振動データに補正される。 Further, when comparing the amount of water, the difference may be calculated by comparing the images. For example, a reference image may be stored as attribute data, and the difference from the reference image may be extracted. For example, as a reference image, an image of a container containing 500 ml of water is prepared, and the reference image is compared with the image of the container containing 200 ml of water taken, and the difference from the reference image, in this case, , The difference that the amount of 300 ml is different may be extracted. Then, the vibration data is corrected according to the extracted difference.
　物体により、属性データとして記憶されているデータの種類が異なるようにしても良い。物体として、数を数えることができるような物体の属性データは、個数の情報とし、量を比較できるような物体の属性データは、基準画像とするようにしても良い。 The type of data stored as attribute data may differ depending on the object. The attribute data of an object whose number can be counted as an object may be used as information on the number of objects, and the attribute data of an object whose quantity can be compared may be used as a reference image.
　ステップＳ１４において、差分判定部４５により、差分があると判定された場合、ステップＳ１５に処理は進められる。差分判定部４５は、差分があると判定した場合、差分があるとの情報と、振動データを、補正処理部４８に供給し、属性パラメータ取得部２２５に、属性パラメータの取得を指示する。 If the difference determination unit 45 determines in step S14 that there is a difference, the process proceeds to step S15. When the difference determination unit 45 determines that there is a difference, the difference determination unit 45 supplies the information that there is a difference and the vibration data to the correction processing unit 48, and instructs the attribute parameter acquisition unit 225 to acquire the attribute parameter.
　ステップＳ１５において、属性パラメータ取得部２２５は、属性パラメータ記憶部４７から、該当する属性パラメータを取得する。例えば、馬（静止）に関連付けられていた振動データが取得されていた場合、属性パラメータとしても、馬（静止）に関連付けられている属性パラメータが取得される。取得された属性パラメータは、補正処理部４８に供給される。 In step S15, the attribute parameter acquisition unit 225 acquires the corresponding attribute parameter from the attribute parameter storage unit 47. For example, when the vibration data associated with the horse (stationary) has been acquired, the attribute parameter associated with the horse (stationary) is also acquired as the attribute parameter. The acquired attribute parameter is supplied to the correction processing unit 48.
　ステップＳ１６において、補正処理部４８は、振動データを、属性パラメータを用いて補正する。例えば、図５を参照して説明したように、２頭の馬が撮影されていた場合、属性パラメータとして“頭数倍”とのパラメータが取得されるため、補正処理部４８では、馬振動データＡを２倍する補正が行われる。 In step S16, the correction processing unit 48 corrects the vibration data using the attribute parameters. For example, as described with reference to FIG. 5, when two horses are photographed, a parameter of "number of horses times" is acquired as an attribute parameter, so that the correction processing unit 48 performs horse vibration data A. Is doubled.
　また例えば、図７、図８を参照して説明したように、２００ｍｌの水が入った容器が撮影された場合、属性パラメータとして“（水量×ａ）倍”とのパラメータが取得されるため、補正処理部４８では、水振動データＡを（２００×ａ）倍する補正が行われる。 Further, for example, as described with reference to FIGS. 7 and 8, when a container containing 200 ml of water is photographed, a parameter of “(water volume × a) times” is acquired as an attribute parameter. The correction processing unit 48 corrects the water vibration data A by (200 × a) times.
　図４を参照するに、データベースには水に関するデータとして、水（振る）と水（注ぐ）の２つのデータが記憶されている。また、この２つのデータは、アクションも関連付けられており、ユーザのアクションがあったときに、振動が開始される。よって、水（振る）と水（注ぐ）の２つのデータのどちらを読み出すかは、水が入った容器が撮影された時点では決定できない。 With reference to FIG. 4, two data, water (shaking) and water (pouring), are stored in the database as data related to water. In addition, these two data are also associated with an action, and vibration is started when there is a user action. Therefore, which of the two data, water (shaking) and water (pouring), cannot be read at the time when the container containing water is photographed.
　このようなデータの場合、２つのデータを読み出し、それぞれ補正処理し、振動データを生成しておくようにしても良い。または、図１０で示したフローチャートの流れによる処理の順番を入れ替え、ユーザのアクションがあったあと、すなわち、水（振る）と水（注ぐ）の２つのデータのうちのどちらを読み出せば良いかが判定された時点で、該当するデータが読み出され、補正処理され、振動データが生成されるようにしても良い。 In the case of such data, two data may be read out, corrected for each, and vibration data may be generated. Alternatively, the order of processing according to the flow chart shown in FIG. 10 should be changed, and after the user has taken an action, that is, which of the two data, water (shaking) and water (pouring), should be read. When is determined, the corresponding data may be read out, corrected, and vibration data may be generated.
　ここでは、図１０に示した処理の流れで処理が行われるとし、２つのデータが読み出され、それぞれ補正処理部４８により補正がされるとして説明を続ける。 Here, it is assumed that the processing is performed according to the processing flow shown in FIG. 10, two data are read out, and the correction processing unit 48 corrects each of the two data.
　ステップＳ１６において補正処理部４８により振動データが補正された場合、補正された振動データは、振動生成提示部４９に供給され、処理は、ステップＳ１７に進められる。 When the vibration data is corrected by the correction processing unit 48 in step S16, the corrected vibration data is supplied to the vibration generation presentation unit 49, and the processing proceeds to step S17.
　ステップＳ１７の処理には、ステップＳ１４において、物体情報についての差分はないと判定された場合にも来る。差分判定部４５は、差分はないと判定した場合、差分はないとの情報と振動データを、補正処理部４８に供給する。補正処理部４８は、差分がないとの情報を受け取った場合、振動データを、振動生成提示部４９に供給する。 The process of step S17 also comes when it is determined in step S14 that there is no difference in the object information. When the difference determination unit 45 determines that there is no difference, the difference determination unit 45 supplies the information that there is no difference and the vibration data to the correction processing unit 48. When the correction processing unit 48 receives the information that there is no difference, the correction processing unit 48 supplies the vibration data to the vibration generation presentation unit 49.
　ステップＳ１７の処理が行われるときには、振動生成提示部４９に、基準振動データ、または補正された振動データが供給されている状態である。ステップＳ１７において、振動生成提示部４９は、ユーザからのアクションが必要な物体か否かを判定する。この判定は、アクションとして記載されているデータがあるか否かが判定されることで行われる。 When the process of step S17 is performed, the reference vibration data or the corrected vibration data is supplied to the vibration generation presentation unit 49. In step S17, the vibration generation presentation unit 49 determines whether or not the object requires an action from the user. This determination is made by determining whether or not there is data described as an action.
　ステップＳ１７において、ユーザからのアクションが必要な物体であると判定された場合、ステップＳ１８に処理が進められる。ステップＳ１８において、振動生成提示部４９は、ユーザからのアクションがあったか否かを判定する。 If it is determined in step S17 that the object requires an action from the user, the process proceeds to step S18. In step S18, the vibration generation presentation unit 49 determines whether or not there has been an action from the user.
　例えば、振動生成提示部４９が、水（振る）の振動データと水（注ぐ）の振動データが保持している状態の場合、水（振る）の振動データは、アクションとして上下振動が行われたときに提示すると設定され、水（注ぐ）の振動データは、アクションとして傾けるという動作が行われたときに提示すると設定されている。このような場合、上下振動または傾けるといった動作がされたか否かが判定される。 For example, when the vibration generation presentation unit 49 holds the vibration data of water (shaking) and the vibration data of water (pouring), the vibration data of water (shaking) is vibrated up and down as an action. It is set to be presented at times, and the vibration data of water (pouring) is set to be presented when the action of tilting is performed as an action. In such a case, it is determined whether or not an operation such as vertical vibration or tilting has been performed.
　ユーザが情報処理装置１０を上下方向に振った場合、振動生成提示部４９は、ユーザからのアクションがあったと判定し、処理をステップＳ１９に進める。 When the user shakes the information processing device 10 in the vertical direction, the vibration generation presentation unit 49 determines that there is an action from the user, and proceeds to the process in step S19.
　ステップＳ１９において、振動生成提示部４９は、振動データに基づき、振動を生成し、ユーザに提示する。例えば、ユーザが情報処理装置１０を上下方向に振った場合には、水（振る）の振動データに基づき、振動が生成され、ユーザに提示される。 In step S19, the vibration generation presentation unit 49 generates vibration based on the vibration data and presents it to the user. For example, when the user shakes the information processing device 10 in the vertical direction, vibration is generated based on the vibration data of water (shaking) and presented to the user.
　一方、ステップＳ１７において、ユーザからのアクションが必要な物体ではないと判定された場合、ステップＳ１９に処理が進められる。例えば、振動生成提示部４９が、馬（静止）の振動データを保持している状態の場合、馬（静止）の振動データは、アクションが規定されていないため、ユーザからのアクションが必要な物体ではないと判定される。 On the other hand, if it is determined in step S17 that the object does not require an action from the user, the process proceeds to step S19. For example, when the vibration generation presentation unit 49 holds the vibration data of the horse (stationary), the vibration data of the horse (stationary) is an object that requires an action from the user because the action is not specified. It is judged that it is not.
　よって、このような場合、馬が撮影された時点で、その馬に関する振動が、馬（静止）の振動データから生成され、ユーザに提示される。 Therefore, in such a case, when the horse is photographed, the vibration related to the horse is generated from the vibration data of the horse (stationary) and presented to the user.
　このように、撮像された物体に関する振動が、ユーザに提示されるため、ユーザは、物体の振動を擬似的に体感することができる。また、物体に応じて、撮像された時点で振動を提示することもできるし、物体に関連する何らかのアクションが行われた時点で、振動を提示することもできる。同一の物体であっても、アクションに応じた振動をユーザに提示することができる。 In this way, the vibration related to the imaged object is presented to the user, so that the user can experience the vibration of the object in a simulated manner. In addition, depending on the object, vibration can be presented at the time of imaging, or vibration can be presented at the time when some action related to the object is performed. Even if it is the same object, it is possible to present the vibration according to the action to the user.
　また同一の物体であっても、個数や量により、異なる振動を提示することができる。よって、ユーザは、個数や量の違いによる振動の違いを体感することができる。このような個数や量の違いを体感させるためのデータとしては、基準となる基準振動データと、属性パラメータを用意しておけば良く、異なる個数や量毎にデータを用意しておかなくても良いため、記憶しておくデータ量を少なくすることができる。また、様々な個数や量に対応することができる。 Even for the same object, different vibrations can be presented depending on the number and quantity. Therefore, the user can experience the difference in vibration due to the difference in the number and amount. As data for experiencing such a difference in number and quantity, it is sufficient to prepare reference vibration data and attribute parameters as a reference, and it is not necessary to prepare data for different numbers and quantities. Because it is good, the amount of data to be stored can be reduced. Moreover, it is possible to correspond to various numbers and quantities.
　＜情報処理システムの構成＞
　上記した情報処理装置１０が行う処理の一部を、サーバで行う構成とすることもできる。例えば、図１１に示すように、情報処理装置２０１とサーバ２０２から構成される情報処理システム２００において、上記したユーザに振動を提示する処理を実行することもできる。なお、上記した情報処理装置１０を例に挙げた説明は、以下に説明する情報処理システム２００に対しても適用できるため、重複する説明は適宜省略するが、以下に記載されていなくても、上記した説明は、適宜適用できるものとして説明を続ける。 <Configuration of information information system>
A part of the processing performed by the information processing device 10 described above may be performed by the server. For example, as shown in FIG. 11, in the information processing system 200 including the information processing device 201 and the server 202, the above-mentioned processing of presenting vibration to the user can be executed. Since the above description using the information processing device 10 as an example can be applied to the information processing system 200 described below, duplicate description will be omitted as appropriate, but even if it is not described below, it may be omitted. The above description will continue as if it were applicable as appropriate.
　図１１に示した情報処理装置２０１は、撮像部２１１、物体認識部２１２、通信部２１３、および振動生成提示部２１４を備える。サーバ２０２は、通信部２２１、音声データ取得部２２２、音声データ記憶部２２３、差分判定部２２４、属性パラメータ取得部２２５、属性パラメータ記憶部２２６、および振動データ生成部２２７を備える。 The information processing device 201 shown in FIG. 11 includes an imaging unit 211, an object recognition unit 212, a communication unit 213, and a vibration generation presentation unit 214. The server 202 includes a communication unit 221, a voice data acquisition unit 222, a voice data storage unit 223, a difference determination unit 224, an attribute parameter acquisition unit 225, an attribute parameter storage unit 226, and a vibration data generation unit 227.
　情報処理装置２０１の撮像部２１１は、物体を撮像するカメラを含む構成とされている。撮像部２１１で撮像された物体画像のデータは、物体認識部２１２に供給される。物体認識部２１２は、撮像された画像内の物体を認識する。認識された物体の情報（認識物体情報）は、通信部２１３に供給される。物体認識情報は、例えば、所定の物体、その物体の個数や量などの情報である。 The imaging unit 211 of the information processing device 201 is configured to include a camera that images an object. The data of the object image captured by the image pickup unit 211 is supplied to the object recognition unit 212. The object recognition unit 212 recognizes an object in the captured image. The recognized object information (recognized object information) is supplied to the communication unit 213. The object recognition information is, for example, information such as a predetermined object and the number and amount of the objects.
　通信部２１３は、サーバ２０２と通信を行い、物体認識情報をサーバ２０２に送信する。また通信部２１３は、サーバ２０２からの振動データを受信し、振動生成提示部２１４に供給する。振動生成提示部２１４は、供給された振動データに基づく振動を生成し、ユーザに提示する。 The communication unit 213 communicates with the server 202 and transmits the object recognition information to the server 202. Further, the communication unit 213 receives the vibration data from the server 202 and supplies it to the vibration generation presentation unit 214. The vibration generation presentation unit 214 generates vibration based on the supplied vibration data and presents it to the user.
　なお、ここでは、情報処理装置２０１が、物体認識部２１２を備えるとして説明を続けるが、物体認識部２１２は、サーバ２０２が有する構成とすることもできる。またそのような構成とした場合、撮像部２１１で撮像された画像が、通信部２１３を介して、サーバ２０２に送信される構成とされる。 Although the description continues here assuming that the information processing device 201 includes the object recognition unit 212, the object recognition unit 212 may be configured to have the server 202. Further, in such a configuration, the image captured by the imaging unit 211 is transmitted to the server 202 via the communication unit 213.
　サーバ２０２の通信部２２１は、情報処理装置２０１と通信を行い、物体認識情報を受信する。また、通信部２２１は、振動データを情報処理装置２０１に送信する。 The communication unit 221 of the server 202 communicates with the information processing device 201 and receives the object recognition information. Further, the communication unit 221 transmits the vibration data to the information processing device 201.
　通信部２２１により受信された物体認識情報は、音声データ取得部２２２と差分判定部２２４に供給される。図１１に示した情報処理システム２００においては、音声データを用いて振動データを生成する場合を例に挙げて説明する。音声データから振動データを生成する場合、音声データ取得部２２２がサーバ２０２に設けられる。 The object recognition information received by the communication unit 221 is supplied to the voice data acquisition unit 222 and the difference determination unit 224. In the information processing system 200 shown in FIG. 11, a case where vibration data is generated using voice data will be described as an example. When generating vibration data from voice data, a voice data acquisition unit 222 is provided in the server 202.
　音声データ取得部２２２は、音声データ記憶部２２３に記憶されている音声データであり、認識物体情報に基づく音声データを取得する。音声データ記憶部２２３には、物体と、その物体に関する音声データが記憶されている。また、その音声データは、物体の個数や量などと関連付けられて記憶されている。 The voice data acquisition unit 222 is voice data stored in the voice data storage unit 223, and acquires voice data based on the recognized object information. The voice data storage unit 223 stores an object and voice data related to the object. Further, the voice data is stored in association with the number and quantity of objects.
　例えば認識物体情報が、認識された物体として馬である場合、音声データ取得部２２２は、馬が疾走しているときの音（例えば足音や鳴き声）を表す音声データを、音声データ記憶部２２３から読み出す。また、その読み出される音声データは、例えば、馬１頭という物体の個数と関連付けられて記憶されているデータである。 For example, when the recognized object information is a horse as a recognized object, the voice data acquisition unit 222 transmits voice data representing a sound (for example, footsteps or barks) when the horse is running from the voice data storage unit 223. read out. Further, the read voice data is, for example, data stored in association with the number of objects such as one horse.
　音声データ取得部２２２により取得された音声データは、差分判定部２２４に供給される。差分判定部２２４には、通信部２２１を介して情報処理装置２０１からの物体認識情報も供給される。差分判定部２２４は、物体認識情報が表す物体の個数や量（以下、個数を例に挙げて説明する）と、音声データに関連付けられている物体の個数の差分を判定する。 The voice data acquired by the voice data acquisition unit 222 is supplied to the difference determination unit 224. Object recognition information from the information processing device 201 is also supplied to the difference determination unit 224 via the communication unit 221. The difference determination unit 224 determines the difference between the number and amount of objects represented by the object recognition information (hereinafter, the number will be described as an example) and the number of objects associated with the voice data.
　例えば、物体認識情報が表す物体が馬であり、その物体の個数が２頭であり、音声データに関連付けられている個数が１頭である場合、１頭分の差分があると判定される。 For example, if the object represented by the object recognition information is a horse, the number of the objects is two, and the number associated with the voice data is one, it is determined that there is a difference for one horse.
　差分判定部２２４は、音声データ取得部２２２から供給された音声データを、振動データ生成部２２７に供給（転送）する。また、差分判定部２２４は、差分があると判定した場合、属性パラメータ取得部２２５に属性パラメータを取得するように指示を出す。属性パラメータ取得部２２５は、属性パラメータ記憶部２２６から属性パラメータを取得する。 The difference determination unit 224 supplies (transfers) the voice data supplied from the voice data acquisition unit 222 to the vibration data generation unit 227. Further, when the difference determination unit 224 determines that there is a difference, the difference determination unit 224 instructs the attribute parameter acquisition unit 225 to acquire the attribute parameter. The attribute parameter acquisition unit 225 acquires the attribute parameter from the attribute parameter storage unit 226.
　音声データ記憶部２２３に記憶されている音声データは、物体が所定の個数である場合の音声に関するデータである。以下、音声データ記憶部２２３に記憶されている音声データを、適宜、基準音声データと記述する。また基準音声データと関連付けられている物体の個数や量を、基準個数（量）と記述する。 The voice data stored in the voice data storage unit 223 is data related to voice when the number of objects is a predetermined number. Hereinafter, the voice data stored in the voice data storage unit 223 will be appropriately referred to as reference voice data. Further, the number and amount of objects associated with the reference voice data are described as the reference number (quantity).
　物体認識情報が表す物体の個数が、基準個数と異なる場合、基準音声データを、物体認識情報が表す物体の個数に合う音声データに変換するためのパラメータが、属性パラメータである。 When the number of objects represented by the object recognition information is different from the reference number, the parameter for converting the reference voice data into the voice data matching the number of objects represented by the object recognition information is the attribute parameter.
　振動データ生成部２２７は、差分判定部２２４から、差分がないとの情報が供給された場合、差分判定部２２４を介して供給された音声データを、振動データに変換する。振動データ生成部２２７は、差分判定部２２４から、差分があるとの情報が供給された場合、差分判定部２２４を介して供給された音声データを、属性パラメータ取得部２２５から供給される属性パラメータを用いて補正し、補正後の音声データを、振動データに変換する。 When the difference determination unit 224 supplies information that there is no difference, the vibration data generation unit 227 converts the voice data supplied via the difference determination unit 224 into vibration data. When the information that there is a difference is supplied from the difference determination unit 224, the vibration data generation unit 227 uses the audio data supplied via the difference determination unit 224 as the attribute parameter supplied from the attribute parameter acquisition unit 225. Is corrected using, and the corrected audio data is converted into vibration data.
　振動データ生成部２２７により生成された振動データは、通信部２２１に供給される。通信部２２１は、供給された振動データを、情報処理装置２０１に送信する。 The vibration data generated by the vibration data generation unit 227 is supplied to the communication unit 221. The communication unit 221 transmits the supplied vibration data to the information processing device 201.
　このように、図１１に示した情報処理システム２００においては音声データから振動データが生成される。図１１に示した情報処理システム２００においても、図３に示した情報処理装置１０と同じく、振動データを記憶し、振動データを必要に応じて補正するように構成することももちろんできる。 In this way, in the information processing system 200 shown in FIG. 11, vibration data is generated from voice data. Of course, the information processing system 200 shown in FIG. 11 can also be configured to store vibration data and correct the vibration data as necessary, as in the information processing device 10 shown in FIG.
　図１１に示した情報処理システム２００のように、音声データを振動データに変換することで、予め多くの種類の振動データを用意していなくても、多くの振動データを生成することができるようになる。音声データは、例えば、ネット上にある映像から取得することができる。映像から取得することで、所定の物体と、その物体の音声を取得することができる。このように、物体とその物体に関する音声データは、物体とその物体に関する振動データよりは多く存在していると考えられ、それらの音声データを、情報処理システム２００では用いることができる。 By converting voice data into vibration data as in the information processing system 200 shown in FIG. 11, many types of vibration data can be generated without preparing many types of vibration data in advance. become. The audio data can be acquired from, for example, a video on the Internet. By acquiring from the video, it is possible to acquire a predetermined object and the sound of the object. As described above, it is considered that the object and the audio data about the object are more present than the vibration data about the object and the object, and the audio data can be used in the information processing system 200.
　また、情報処理装置１０（図３）に、振動データを記憶させるようにした場合、また情報処理装置１０がスマートフォンなどの端末である場合、記憶容量に制限があるため、多くの振動データを記憶できない可能性がある。しかしながら情報処理システム２００のように、サーバ２０２を設け、サーバ２０２側で振動データを生成するための音声データを記憶するような構成とすることで、より多くの音声データを記憶することができるようになり、より多くの振動データを供給することが可能となる。 Further, when the information processing device 10 (FIG. 3) is configured to store vibration data, or when the information processing device 10 is a terminal such as a smartphone, a large amount of vibration data is stored because the storage capacity is limited. It may not be possible. However, as in the information processing system 200, by providing the server 202 and configuring the server 202 side to store the voice data for generating the vibration data, more voice data can be stored. Therefore, it becomes possible to supply more vibration data.
　このようなことから、ここでは音声データから振動データを生成する場合を例に挙げて説明を続ける。 For this reason, the explanation will be continued here by taking the case of generating vibration data from voice data as an example.
　＜データベースの構成例＞
　図１２に音声データ記憶部２２３と属性パラメータ記憶部２２６のデータベース構成例を示す。なお、図１１では、説明のため音声データ記憶部２２３と属性パラメータ記憶部２２６を別々に図示したが、図１２に示すように１つのデータベースとすることができ、図１２では、１つのデータベースとした場合の構成例を示す。 <Database configuration example>
FIG. 12 shows a database configuration example of the voice data storage unit 223 and the attribute parameter storage unit 226. Although the voice data storage unit 223 and the attribute parameter storage unit 226 are shown separately in FIG. 11 for the sake of explanation, they can be combined into one database as shown in FIG. 12, and in FIG. 12, one database is used. An example of the configuration is shown below.
　図１２に示したデータベースは、基本的に、図４に示したデータベースと同様の構成を有するため、図４を参照した説明と重複する説明は適宜省略する。 Since the database shown in FIG. 12 basically has the same configuration as the database shown in FIG. 4, the description overlapping with the description with reference to FIG. 4 will be omitted as appropriate.
　音声データ記憶部２２３は、物体名、アクション、音声データ、および属性データが関連付けられて記憶されている。また、属性パラメータ記憶部２２６は、少なくとも物体名と属性パラメータが関連付けられて記憶されている。 The voice data storage unit 223 stores the object name, the action, the voice data, and the attribute data in association with each other. Further, the attribute parameter storage unit 226 stores at least the object name and the attribute parameter in association with each other.
　物体名の欄には、物体名が記載される。アクションの欄には、振動を提示するときに、ユーザからの何らかのアクションが必要な場合、そのアクションが記載される。音声データの欄には、提示する音声データ（基準音声データ）の情報が記憶される。音声データの情報としては、音の大きさ、周波数、ピッチ、時間、余韻などがある。 The object name is entered in the object name column. In the action column, if any action is required from the user when presenting the vibration, that action is described. Information on the presented voice data (reference voice data) is stored in the voice data column. Information on voice data includes loudness, frequency, pitch, time, and afterglow.
　属性データの欄には、物体の個数や量に関するデータや、基準となる物体の状態が撮像された画像（基準画像）が記載される。 In the attribute data column, data on the number and amount of objects and an image (reference image) in which the state of the reference object is captured are described.
　属性パラメータ記憶部２２６の属性パラメータの欄には、属性データに記載されている物体の個数や量以外の個数や量以外のときに、音声データを補正するための情報が記載されている。 In the attribute parameter column of the attribute parameter storage unit 226, information for correcting the voice data is described when the number or amount is other than the number or amount of objects described in the attribute data.
　音声データの欄に記載されている音声データを、基準音声データとした場合、基準音声データは、属性データの欄に記載されている物体の個数のときの音声を生成するためのデータである。属性データの欄に記載されている物体の個数以外の個数のときの音声データを生成するための情報が、属性パラメータである。この補正された音声データから振動データが生成される。 When the voice data described in the voice data column is used as the reference voice data, the reference voice data is data for generating voice when the number of objects is the number of objects described in the attribute data column. The information for generating the voice data when the number of objects is other than the number of objects described in the attribute data column is the attribute parameter. Vibration data is generated from this corrected voice data.
　例えば、物体名が馬（静止）、アクションが空欄、音声データが馬音声データＡ、属性データが１頭という情報が関連付けられている。これらのデータは、図５のＡに示したように、静止している馬１０１のときの基準音声データが、馬音声データＡであることを意味している。また、馬音声データＡは、ユーザのアクションによらず、例えば、静止している馬の画像が撮像された時点で振動データに変換され、振動の提示が開始される。 For example, information that the object name is horse (stationary), the action is blank, the voice data is horse voice data A, and the attribute data is one horse is associated. These data mean that, as shown in A of FIG. 5, the reference voice data when the horse 101 is stationary is the horse voice data A. Further, the horse voice data A is converted into vibration data when, for example, an image of a stationary horse is captured, regardless of the user's action, and the presentation of vibration is started.
　また属性パラメータとして、音量１．３倍という情報が記載されている。この場合、基準音声データを１．３倍することで、頭数分の音声データが生成される。例えば、図５のＢに示したように、情報処理装置１０により、２頭の静止している馬１０２と馬１０３が撮像され、表示部１６に表示された場合、馬音声データＡに、１．３を乗算することで、２頭の馬がいるときの音声データが生成され、その音声データが振動データに変換され、情報処理装置１０によりユーザに振動が提示される。 Also, as an attribute parameter, information that the volume is 1.3 times is described. In this case, by multiplying the reference voice data by 1.3, voice data for the number of heads is generated. For example, as shown in FIG. 5B, when two stationary horses 102 and 103 are imaged by the information processing device 10 and displayed on the display unit 16, the horse voice data A contains 1 By multiplying by 3, voice data when there are two horses is generated, the voice data is converted into vibration data, and the information processing device 10 presents vibration to the user.
　図１２を参照するに例えば、物体名が馬（疾走）、アクションが空欄、音声データが馬音声データＢ、属性データが１頭という情報が関連付けられている。これらのデータは、図６のＡに示したように、疾走している馬１１１のときの基準音声データが、馬音声データＢであることを意味している。また、馬音声データＢは、ユーザのアクションによらず、例えば、疾走している馬の画像が撮像された時点で振動データに変換され、振動の提示が開始される。 With reference to FIG. 12, for example, information that the object name is a horse (sprinting), the action is blank, the voice data is horse voice data B, and the attribute data is one is associated. These data mean that, as shown in A of FIG. 6, the reference voice data at the time of the running horse 111 is the horse voice data B. Further, the horse voice data B is converted into vibration data when, for example, an image of a running horse is captured, regardless of the user's action, and the presentation of vibration is started.
　また属性パラメータとして、２倍という情報が記載されている。この場合、基準音声データを２倍することで、頭数分の音声データが生成される。例えば、図６のＢに示したように、情報処理装置１０により、２頭の疾走している馬１１２と馬１１３が撮像され、表示部１６に表示された場合、馬音声データＢに、２を乗算することで、２頭の馬が疾走しているときの音声データが生成され、その補正後の音声データが、振動データに変換され、情報処理装置１０によりユーザに振動が提示される。 Also, as an attribute parameter, the information of double is described. In this case, by doubling the reference voice data, voice data for the number of heads is generated. For example, as shown in B of FIG. 6, when two running horses 112 and 113 are imaged by the information processing device 10 and displayed on the display unit 16, the horse voice data B contains 2 By multiplying, voice data when two horses are running is generated, the corrected voice data is converted into vibration data, and the information processing device 10 presents vibration to the user.
　図１２を参照するに例えば、物体名が水（振る）、アクションが上下振動、音声データが水音声データＡ、属性データが５００ｍｌという情報が関連付けられている。これらのデータは、図７のＡに示したように、ペットボトルなどの容器１３１に５００ｍｌ（満水の状態）のときに、容器１３１が上下に振られたときの基準音声データが、水音声データＡであることを意味している。 With reference to FIG. 12, for example, information that the object name is water (shaking), the action is vertical vibration, the voice data is water voice data A, and the attribute data is 500 ml is associated. As shown in A of FIG. 7, these data are based on the water voice data when the container 131 is shaken up and down when the container 131 such as a PET bottle is 500 ml (full state). It means that it is A.
　また、水音声データＡは、ユーザが、アクションとして上下振動、例えば情報処理装置１０を上下に揺らしたり、情報処理装置１０の画面を指でタッチしながら上下に動かしたりするなどのアクションがされたときに、水が容器１３１内で上下に振動しているときの音のデータである。そして、水音声データＡが振動データに変換されることで、振動が、ユーザに提示される。 Further, in the water voice data A, the user takes an action such as vertical vibration, for example, shaking the information processing device 10 up and down, or moving the screen of the information processing device 10 up and down while touching the screen with a finger. Sometimes, it is sound data when water vibrates up and down in the container 131. Then, the water voice data A is converted into vibration data, so that the vibration is presented to the user.
　また属性パラメータとして、高音増という情報が記載されている。この場合、基準音声データの周波数成分のうち高音領域の音を増すことで、水量に合った音声データが生成される。例えば、図７のＢに示したように、情報処理装置１０により、水が２００ｍｌ入っている容器１３２が撮像される。そして、図８のＡに示したように、ユーザが表示部１６（タッチパネル）上を、指でタッチしながら上下に動かすと、振動の提示が開始される。この場合、２００ｍｌの水が、容器１３２内で上下に動くときの音の音声データが、水音声データＡの高音部分を増やす補正がされることで生成され、その音声データに基づき変換された振動データに基づく振動が、ユーザに提示される。 Also, as an attribute parameter, information about treble increase is described. In this case, by increasing the sound in the treble region among the frequency components of the reference voice data, voice data corresponding to the amount of water is generated. For example, as shown in B of FIG. 7, the information processing apparatus 10 images a container 132 containing 200 ml of water. Then, as shown in FIG. 8A, when the user moves the display unit 16 (touch panel) up and down while touching it with a finger, the presentation of vibration is started. In this case, the sound data when 200 ml of water moves up and down in the container 132 is generated by correcting to increase the high-pitched part of the water sound data A, and the vibration converted based on the sound data. The vibration based on the data is presented to the user.
　図１２を参照するに例えば、物体名が水（注ぐ）、アクションが傾ける、音声データが水音声データＢ、属性データが５００ｍｌという情報が関連付けられている。これらのデータは、図７のＡに示したように、ペットボトルなどの容器１３１に５００ｍｌ（満水の状態）のときに、容器１３１が傾けられたとき（水が容器１３１からコップなどに注がれるとき）の基準音声データが、水音声データＢであることを意味している。 With reference to FIG. 12, for example, information that the object name is water (pouring), the action is tilted, the voice data is water voice data B, and the attribute data is 500 ml is associated. As shown in A of FIG. 7, these data are obtained when the container 131 such as a PET bottle is 500 ml (full state) and the container 131 is tilted (water is poured from the container 131 into a cup or the like). It means that the reference audio data of (when) is the water audio data B.
　また、水音声データＢは、ユーザが、アクションとして傾ける、例えば情報処理装置１０を傾けたり、情報処理装置１０の画面を指でタッチしながら容器１３１の口から水が出るような方向に動かしたりするなどのアクションがされたときに、水が容器１３１からコップなどに注がれるときの音のデータである。そして、水音声データＢが振動データに変換されることで、振動が、ユーザに提示される。 Further, the water voice data B can be tilted as an action by the user, for example, tilting the information processing device 10 or moving the water voice data B in a direction in which water comes out from the mouth of the container 131 while touching the screen of the information processing device 10 with a finger. This is sound data when water is poured from a container 131 into a cup or the like when an action such as is performed. Then, the water voice data B is converted into vibration data, so that the vibration is presented to the user.
　また属性パラメータとして、高音増という情報が記載されている。この場合、基準音声データの周波数成分のうちの高音領域の音を増すことで、水量に合った音声データが生成される。例えば、図７のＢに示したように、情報処理装置１０により、水が２００ｍｌ入っている容器１３２が撮像される。そして、図８のＢに示したように、ユーザが情報処理装置１０を傾けると、振動の提示が開始される。この場合、２００ｍｌの水が、容器１３２から注がれるときの音の音声データが、水音声データＢの高音領域を増やす補正がされることで生成され、その音声データに基づき変換された振動データに基づく振動が、ユーザに提示される。 Also, as an attribute parameter, information about treble increase is described. In this case, by increasing the sound in the high frequency region of the frequency components of the reference voice data, voice data matching the amount of water is generated. For example, as shown in B of FIG. 7, the information processing apparatus 10 images a container 132 containing 200 ml of water. Then, as shown in FIG. 8B, when the user tilts the information processing device 10, the presentation of vibration is started. In this case, the vibration data of the sound when 200 ml of water is poured from the container 132 is generated by correcting to increase the treble region of the water voice data B, and the vibration data is converted based on the voice data. The vibration based on is presented to the user.
　図１２を参照するに例えば、物体名が車（アイドリング）、アクションがタップ、音声データが車音声データＡ、属性データが１台という情報が関連付けられている。これらのデータは、図９のＡに示したように、アイドリングしている車１５１のときの基準音声データが、車音声データＡであることを意味している。また、車音声データＡは、振動データに変換され、ユーザが情報処理装置１０をタップしたときに、その振動データに基づく振動の提示が開始される。 With reference to FIG. 12, for example, information that the object name is a car (idling), the action is a tap, the voice data is the car voice data A, and the attribute data is one is associated. These data mean that, as shown in A of FIG. 9, the reference voice data when the vehicle is idling 151 is the car voice data A. Further, the vehicle voice data A is converted into vibration data, and when the user taps the information processing device 10, the presentation of vibration based on the vibration data is started.
　また属性パラメータとして、音量２倍、走行時３倍という情報が記載されている。この場合、基準音声データの音量を２倍することで、アイドリング時の音声データが生成され、３倍することで、走行時の音声データが生成される。例えば、図９のＢに示したように、情報処理装置１０により、２台のアイドリングしている車１５２と車１５３が撮像され、表示部１６に表示された場合、車音声データＡの音量を２倍にすることで、２台の車がアイドリングしているときの音声データが生成され、その音声データが振動データに変換されることで、情報処理装置１０により振動がユーザに提示される。 Also, as attribute parameters, information that the volume is doubled and the running time is tripled is described. In this case, by doubling the volume of the reference voice data, the voice data at the time of idling is generated, and by doubling the volume, the voice data at the time of traveling is generated. For example, as shown in B of FIG. 9, when two idling cars 152 and 153 are imaged by the information processing device 10 and displayed on the display unit 16, the volume of the car voice data A is set. By doubling, voice data when the two cars are idling is generated, and the voice data is converted into vibration data, so that the information processing device 10 presents the vibration to the user.
　このように、データベースとして、基準音声データや、その基準音声データを補正するためのパラメータなどが、サーバ２０２には記憶されている。また、基準音声データに関する情報として、物体の個数や量を含む物体の状態に関する情報を記憶しておき、その情報との差分が抽出され、その差分に応じた音声データに変換され、さらにその音声データが振動データに変換される。よって、ユーザに、物体の状態に応じた振動を提示することができる。 In this way, as a database, the reference voice data and the parameters for correcting the reference voice data are stored in the server 202. Further, as information on reference audio data, information on the state of an object including the number and quantity of objects is stored, a difference from the information is extracted, converted into audio data according to the difference, and the audio is further generated. The data is converted to vibration data. Therefore, it is possible to present the user with vibration according to the state of the object.
　また、基準音声データに関する情報として、物体の個数や量を含む物体の状態に関する画像を基準画像として用意しておき、その基準画像との撮像された画像との差分が抽出され、その差分に応じた音声データに変換され、さらにその音声データが振動データに変換される。よって、ユーザに、物体の状態に応じた振動を提示することができる。 Further, as information on the reference audio data, an image relating to the state of the object including the number and quantity of the objects is prepared as the reference image, and the difference between the reference image and the captured image is extracted, and according to the difference. It is converted into audio data, and the audio data is further converted into vibration data. Therefore, it is possible to present the user with vibration according to the state of the object.
　＜情報処理装置の動作＞
　図１３のフローチャートを参照し、上記したような振動を提示する情報処理システム２００の動作について説明する。 <Operation of information processing device>
The operation of the information processing system 200 that presents the vibration as described above will be described with reference to the flowchart of FIG.
　ステップＳ１０１において、情報処理装置２０１の撮像部２１１により画像が撮像される。ユーザが、情報処理装置２０１を保持している状態で、所定の物体にカメラを向け、撮像が行われる。 In step S101, an image is captured by the imaging unit 211 of the information processing device 201. While the user holds the information processing device 201, the user points the camera at a predetermined object to perform imaging.
　ステップＳ１０２において、物体認識部２１２は、撮像部２１１で撮像された画像を解析し、画像内の物体、その物体の個数や量、状態などを検出し、それらの情報を含む物体認識情報を生成する。ステップＳ１０３において、物体認識部２１２により生成された認識物体情報は、通信部２１３を介してサーバ２０２に送信される。 In step S102, the object recognition unit 212 analyzes the image captured by the image pickup unit 211, detects an object in the image, the number, amount, state, etc. of the object, and generates object recognition information including the information. To do. In step S103, the recognized object information generated by the object recognition unit 212 is transmitted to the server 202 via the communication unit 213.
　ステップＳ１２１において、サーバ２０２の通信部２２１は、情報処理装置２０１からの認識物体情報を受信する。受信された認識物体情報は、音声データ取得部２２２と差分判定部２２４に供給される。 In step S121, the communication unit 221 of the server 202 receives the recognized object information from the information processing device 201. The received recognized object information is supplied to the voice data acquisition unit 222 and the difference determination unit 224.
　ステップＳ１２２において、音声データ取得部２２２は、音声データ記憶部２２３から、認識物体情報が示す物体（物体名）に関連付けられているデータを読み出す。読み出されるデータは、図１２を参照して説明したように、アクション、音声データ、属性データである。読み出されたデータは、差分判定部２２４に供給される。 In step S122, the voice data acquisition unit 222 reads the data associated with the object (object name) indicated by the recognized object information from the voice data storage unit 223. The data to be read is action, voice data, and attribute data, as described with reference to FIG. The read data is supplied to the difference determination unit 224.
　ステップＳ１２３において、差分判定部２２４は、物体情報についての差分があるか否かを判定する。差分判定部２２４は、音声データ取得部２２２から供給された属性データと、物体認識部２１２から供給された認識物体情報を参照し、属性データで示される個数（量）と、認識物体情報で示される個数（量）を比較し、差分があるか否かを判定する。 In step S123, the difference determination unit 224 determines whether or not there is a difference in the object information. The difference determination unit 224 refers to the attribute data supplied from the voice data acquisition unit 222 and the recognized object information supplied from the object recognition unit 212, and indicates the number (quantity) indicated by the attribute data and the recognized object information. Compare the number (quantity) of the data and determine if there is a difference.
　例えば、図５を参照して説明したように、２頭の馬が撮影された場合、認識物体情報で示される個数が２であり、属性データが示す個数が１であるため、差分があるとの判定が出される。また、１頭の馬が撮影された場合、認識物体情報で示される個数が１であり、属性データが示す個数が１であるため、差分はないとの判定が出される。 For example, as described with reference to FIG. 5, when two horses are photographed, the number indicated by the perceptible object information is 2, and the number indicated by the attribute data is 1, so that there is a difference. Is issued. Further, when one horse is photographed, the number indicated by the recognized object information is 1, and the number indicated by the attribute data is 1, so it is determined that there is no difference.
　ステップＳ１２３において、差分判定部２２４により、差分があると判定された場合、ステップＳ１２４に処理は進められる。差分判定部２２４は、差分があると判定した場合、差分があるとの情報と、音声データを、振動データ生成部２２７に供給し、属性パラメータ取得部２２５に、属性パラメータの取得を指示する。 If the difference determination unit 224 determines in step S123 that there is a difference, the process proceeds to step S124. When the difference determination unit 224 determines that there is a difference, the difference determination unit 224 supplies the information that there is a difference and the voice data to the vibration data generation unit 227, and instructs the attribute parameter acquisition unit 225 to acquire the attribute parameter.
　ステップＳ１２４において、属性パラメータ取得部２２５は、属性パラメータ記憶部２２６から、該当する属性パラメータを取得する。例えば、馬（静止）に関連付けられていた音声データが取得されていた場合、属性パラメータとしても、馬（静止）に関連付けられている属性パラメータが取得される。取得された属性パラメータは、振動データ生成部２２７に供給される。 In step S124, the attribute parameter acquisition unit 225 acquires the corresponding attribute parameter from the attribute parameter storage unit 226. For example, when the voice data associated with the horse (stationary) has been acquired, the attribute parameter associated with the horse (stationary) is also acquired as the attribute parameter. The acquired attribute parameters are supplied to the vibration data generation unit 227.
　ステップＳ１２５において、振動データ生成部２２７は、音声データを、属性パラメータを用いて補正する。例えば、図５を参照して説明したように、２頭の馬が撮影されていた場合、属性パラメータとして“音量１．３倍”とのパラメータが取得されるため、振動データ生成部２２７では、馬音声データＡの音量を１．３倍する補正が行われる。 In step S125, the vibration data generation unit 227 corrects the voice data using the attribute parameters. For example, as described with reference to FIG. 5, when two horses are photographed, a parameter of “volume 1.3 times” is acquired as an attribute parameter, so that the vibration data generation unit 227 may use the vibration data generation unit 227. The volume of the horse voice data A is corrected by 1.3 times.
　また例えば、図７、図８を参照して説明したように、２００ｍｌの水が入った容器が撮影された場合、属性パラメータとして“高音増”とのパラメータが取得されるため、振動データ生成部２２７では、水音声データＡの高音領域を増す補正が行われる。 Further, for example, as described with reference to FIGS. 7 and 8, when a container containing 200 ml of water is photographed, a parameter of “treble increase” is acquired as an attribute parameter, so that the vibration data generation unit At 227, correction is performed to increase the treble region of the water voice data A.
　図１２を参照するに、データベースには水に関するデータとして、水（振る）と水（注ぐ）の２つのデータが記憶されている。また、この２つのデータは、アクションも関連付けられており、ユーザのアクションがあったときに、振動が開始される。よって、水（振る）と水（注ぐ）の２つのデータのどちらを読み出すかは、認識物体情報が取得された時点では決定できない。 With reference to FIG. 12, two data, water (shaking) and water (pouring), are stored in the database as data related to water. In addition, these two data are also associated with an action, and vibration is started when there is a user action. Therefore, which of the two data, water (shaking) and water (pouring), cannot be read at the time when the perceptible object information is acquired.
　このようなデータの場合、２つの音声データを読み出し、それぞれ補正処理し、振動データを生成し、送信するようにしても良い。または、図１３で示したフローチャートの流れによる処理の順番を入れ替え、ユーザのアクションがあったときに、そのアクションに関する情報が、情報処理装置２０１側から送信されるようにし、その送信された情報を受信した時点、すなわち、水（振る）と水（注ぐ）の２つのデータのうちのどちらを読み出せば良いかが判定された時点で、該当する音声データが読み出され、補正処理され、振動データが生成されるようにしても良い。 In the case of such data, two voice data may be read out, corrected for each, vibration data generated, and transmitted. Alternatively, the order of processing according to the flow chart shown in FIG. 13 is changed so that when there is an action of the user, information about the action is transmitted from the information processing apparatus 201 side, and the transmitted information is transmitted. When it is received, that is, when it is determined which of the two data, water (shaking) and water (pouring), should be read, the corresponding audio data is read, corrected, and vibrated. Data may be generated.
　ここでは、図１３に示した処理の流れで処理が行われるとし、２つのデータが読み出され、それぞれ振動データ生成部２２７により補正がされるとして説明を続ける。 Here, assuming that the processing is performed according to the processing flow shown in FIG. 13, two data are read out and corrected by the vibration data generation unit 227, respectively.
　ステップＳ１２５において、振動データ生成部２２７は、音声データを補正する。そしてステップＳ１２６において、振動データ生成部２２７は、音声データを振動データに変換し、通信部２２１に供給する。通信部２２１は、ステップＳ１２７において、供給された振動データを、情報処理装置２０１に送信する。 In step S125, the vibration data generation unit 227 corrects the voice data. Then, in step S126, the vibration data generation unit 227 converts the voice data into vibration data and supplies it to the communication unit 221. The communication unit 221 transmits the supplied vibration data to the information processing device 201 in step S127.
　ステップＳ１２６の処理には、ステップＳ１２３において、物体情報についての差分はないと判定された場合にも来る。差分判定部２２４は、差分はないと判定した場合、差分はないとの情報と音声データを、振動データ生成部２２７に供給する。振動データ生成部２２７は、差分がないとの情報を受け取った場合、音声データを振動データに変換し、通信部２２１に供給する。通信部２２１は、ステップＳ１２７において、供給された振動データを、情報処理装置２０１に送信する。 The process of step S126 also comes when it is determined in step S123 that there is no difference in the object information. When the difference determination unit 224 determines that there is no difference, the difference determination unit 224 supplies the information that there is no difference and the voice data to the vibration data generation unit 227. When the vibration data generation unit 227 receives the information that there is no difference, the vibration data generation unit 227 converts the voice data into vibration data and supplies it to the communication unit 221. The communication unit 221 transmits the supplied vibration data to the information processing device 201 in step S127.
　サーバ２０２から送信された振動データは、ステップＳ１０４において、情報処理装置２０１の通信部２１３により受信される。受信された振動データは、振動生成提示部２１４に供給される。振動生成提示部２１４は、ステップＳ１０５において、ユーザからのアクションが必要な物体か否かを判定する。この判定は、アクションとして設定されているデータがあるか否かが判定されることで行われる。また、このような判定を行うためのデータも、振動データとともに、サーバ２０２側から送信されてきている。 The vibration data transmitted from the server 202 is received by the communication unit 213 of the information processing device 201 in step S104. The received vibration data is supplied to the vibration generation presentation unit 214. In step S105, the vibration generation presentation unit 214 determines whether or not the object requires an action from the user. This determination is made by determining whether or not there is data set as an action. Further, the data for making such a determination is also transmitted from the server 202 side together with the vibration data.
　ステップＳ１０５において、ユーザからのアクションが必要な物体であると判定された場合、ステップＳ１０６に処理が進められる。ステップＳ１０６において、振動生成提示部２１４は、ユーザからのアクションがあったか否かを判定する。 If it is determined in step S105 that the object requires an action from the user, the process proceeds to step S106. In step S106, the vibration generation presentation unit 214 determines whether or not there has been an action from the user.
　例えば、振動生成提示部２１４が、水（振る）の振動データと水（注ぐ）の振動データを保持している状態の場合、水（振る）の振動データは、アクションとして上下振動が行われたときに提示すると設定され、水（注ぐ）の振動データは、アクションとして傾けるという動作が行われたときに提示すると設定されている。このような場合、上下振動または傾けるといった動作がされたか否かが判定される。 For example, when the vibration generation presentation unit 214 holds the vibration data of water (shaking) and the vibration data of water (pouring), the vibration data of water (shaking) is vibrated up and down as an action. It is set to be presented at times, and the vibration data of water (pouring) is set to be presented when the action of tilting is performed as an action. In such a case, it is determined whether or not an operation such as vertical vibration or tilting has been performed.
　ユーザが情報処理装置２０１を上下方向に振った場合、振動生成提示部２１４は、ユーザからのアクションがあったと判定し、処理をステップＳ１０７に進める。 When the user shakes the information processing device 201 in the vertical direction, the vibration generation presentation unit 214 determines that there is an action from the user, and proceeds to the process in step S107.
　ステップＳ１０７において、振動生成提示部２１４は、振動データに基づき、振動を生成し、ユーザに提示する。例えば、ユーザが情報処理装置２０１を上下方向に振った場合には、水（振る）の音声データに基づき、振動が生成され、ユーザに提示される。 In step S107, the vibration generation presentation unit 214 generates vibration based on the vibration data and presents it to the user. For example, when the user shakes the information processing device 201 in the vertical direction, vibration is generated based on the voice data of water (shaking) and presented to the user.
　一方、ステップＳ１０５において、ユーザからのアクションが必要な物体ではないと判定された場合、ステップＳ１０７に処理が進められる。例えば、振動生成提示部２１４が、馬（静止）の音声データを保持している状態の場合、馬（静止）の音声データは、アクションが規定されていないため、ユーザからのアクションが必要な物体ではないと判定される。 On the other hand, if it is determined in step S105 that the object does not require an action from the user, the process proceeds to step S107. For example, when the vibration generation presentation unit 214 holds the voice data of the horse (stationary), the voice data of the horse (stationary) is an object that requires an action from the user because the action is not specified. It is judged that it is not.
　よって、このような場合、馬が撮影された時点で、その馬に関する振動が、馬（静止）の振動データから生成され、ユーザに提示される。 Therefore, in such a case, when the horse is photographed, the vibration related to the horse is generated from the vibration data of the horse (stationary) and presented to the user.
　このように、撮像された物体に関する振動が、ユーザに提示されるため、ユーザは、物体の振動を擬似的に体感することができる。また、物体に応じて、撮像された時点で振動を提示することもできるし、物体に関連する何らかのアクションが行われた時点で、振動を提示することもできる。同一の物体であっても、アクションに応じた振動をユーザに提示することができる。 In this way, the vibration related to the imaged object is presented to the user, so that the user can experience the vibration of the object in a simulated manner. In addition, depending on the object, vibration can be presented at the time of imaging, or vibration can be presented at the time when some action related to the object is performed. Even if it is the same object, it is possible to present the vibration according to the action to the user.
　また同一の物体であっても、個数や量により、異なる振動を提示することができる。よって、ユーザは、個数や量の違いによる振動の違いを体感することができる。このような個数や量の違いを体感させるためのデータとしては、基準となる基準音声データと、属性パラメータを用意しておけば良く、異なる個数や量毎にデータを用意しておかなくても良いため、記憶しておくデータ量を少なくすることができる。また、様々な個数や量に対応することができる。 Even for the same object, different vibrations can be presented depending on the number and quantity. Therefore, the user can experience the difference in vibration due to the difference in the number and amount. As data for experiencing such a difference in number and quantity, it is sufficient to prepare reference audio data as a reference and attribute parameters, and it is not necessary to prepare data for different numbers and quantities. Because it is good, the amount of data to be stored can be reduced. Moreover, it is possible to correspond to various numbers and quantities.
　＜フィードバックについて＞
　上記した情報処理装置１０や情報処理システム２００により、ユーザに振動が提示されるが、この提示された振動に対してユーザの評価が、フィードバックとして供給される仕組みを設けても良い。また、フィードバックされる仕組みを設けた場合、フィードバックの内容により、振動データが修正されるようにし、よりユーザの好みに合った振動が提示されるようにしても良い。 <Feedback>
The information processing device 10 and the information processing system 200 described above present vibrations to the user, and a mechanism may be provided in which the user's evaluation is supplied as feedback with respect to the presented vibrations. Further, when a feedback mechanism is provided, the vibration data may be corrected according to the content of the feedback, and the vibration more suitable for the user's preference may be presented.
　また上記した情報処理システム２００においては、音声データを振動データに変換するが、この変換がよりユーザの好みに合った振動が提示される変換となるように学習されるようにしても良い。 Further, in the above-mentioned information processing system 200, voice data is converted into vibration data, and this conversion may be learned so that vibration that is more suitable for the user's preference is presented.
　フィードバックは、振動の誤りを定義するフィードバックとすることができる。例えば、情報処理装置１０（２０１）上に、振動に対する良い、悪いといった評価を入力するＧＵＩ（Graphical User Interface）ボタンを表示し、操作されたボタンの情報（良い、悪い）が、フィードバックとして用いられる。また例えば、ユーザが振るといった動作をしているときに、その動作を中断した場合、提示していた振動がユーザの好みではなかったと判定し、その判定結果が、フィードバックとして用いられるようにしても良い。 Feedback can be feedback that defines vibration errors. For example, a GUI (Graphical User Interface) button for inputting an evaluation of good or bad for vibration is displayed on the information processing device 10 (201), and the information (good or bad) of the operated button is used as feedback. .. Further, for example, if the operation is interrupted while the user is shaking, it is determined that the presented vibration is not the user's preference, and the determination result can be used as feedback. good.
　また振動に対するユーザからの具体的なコメントが、フィードバックとして用いられるようにしても良い。また、振動が物体を撮像しているときに提示されるという設定がなされている場合、振動が提示されている最中に物体の撮像をユーザが中止した場合、提示されていた振動はユーザの好みではなかったと判定し、そのような判定結果が、フィードバックされるようにしても良い。 Also, specific comments from the user regarding vibration may be used as feedback. Also, if the vibration is set to be presented when the object is being imaged, and if the user stops imaging the object while the vibration is being presented, the presented vibration will be presented by the user. It may be determined that the object is not preferred, and such a determination result may be fed back.
　またフィードバックは、振動感を修正するようなフィードバックでも良い。例えば、情報処理装置１０（２０１）上に、振動が強い、弱いといった評価を入力するＧＵＩボタンを表示し、操作されたボタンの情報（強い、弱い）が、フィードバックとして用いられる。また例えば、振動に対するユーザからの具体的なコメントが、フィードバックとして用いられるようにしても良い。 The feedback may be feedback that corrects the feeling of vibration. For example, a GUI button for inputting an evaluation such as strong or weak vibration is displayed on the information processing device 10 (201), and the information (strong or weak) of the operated button is used as feedback. Further, for example, a specific comment from the user regarding vibration may be used as feedback.
　また、例えば、物体からの距離が離れていくような動作がされた場合、振動を弱めるような設定を設ける。そのような設定が設けられている場合、振動中の情報処理装置１０（２０１）の端末情報が、フィードバックとして取得されるようにしても良い。例えば、ユーザが振動中に、端末を物体から遠ざけたという端末情報が得られた場合、振動が強かったため、端末を遠ざけた判定し、次回からは少し弱くした振動が提示されるように修正される。 Also, for example, if the movement is such that the distance from the object increases, a setting is provided to weaken the vibration. When such a setting is provided, the terminal information of the information processing device 10 (201) during vibration may be acquired as feedback. For example, if the user obtains terminal information that the terminal has been moved away from an object during vibration, the vibration was strong, so it is determined that the terminal has been moved away, and the vibration that is slightly weakened will be presented from the next time. To.
　フィードバックによる修正するのは、図３に示した情報処理装置１０の場合、振動データであり、図１１に示した情報処理システム２００の場合、音声データである。 The correction by feedback is vibration data in the case of the information processing device 10 shown in FIG. 3, and voice data in the case of the information processing system 200 shown in FIG.
　＜サーバの構成＞
　情報処理システム２００を構成するサーバ２０２は、図１４に示すような構成とすることができる。図１４に示したサーバ２０２は、３つのサーバ２０２－１、サーバ２０２－２、およびサーバ２０２－３から構成されている。 <Server configuration>
The server 202 that constitutes the information processing system 200 can be configured as shown in FIG. The server 202 shown in FIG. 14 is composed of three servers 202-1, server 202-2, and server 202-3.
　サーバ２０２－１は、情報処理装置２０１から物体認識情報が供給され、その物体認識情報で表される物体に関する音声データを、サーバ２０２－２に対してリクエストする。サーバ２０２－２は、音声データを蓄積しているサーバであり、サーバ２０２－１のリクエストに応じて、音声データを読み出し、サーバ２０２－３に供給する。 The server 202-1 is supplied with object recognition information from the information processing device 201, and requests the server 202-2 for voice data regarding the object represented by the object recognition information. The server 202-2 is a server that stores voice data, reads voice data in response to a request from server 202-1, and supplies the voice data to server 202-3.
　サーバ２０２－３は、音声データを振動データに変換するサーバであり、サーバ２０２－２から音声データが供給された場合、その音声データを、振動データに変換し、情報処理装置２０１に供給する。 The server 202-3 is a server that converts voice data into vibration data, and when voice data is supplied from server 202-2, the voice data is converted into vibration data and supplied to the information processing device 201.
　このような構成とした場合、サーバ２０２－１とサーバ２０２－３は、独自のＡＰＩ（Application Programming Interface）サーバとすることができる。またサーバ２０２－２は、パブリックなＡＰＩサーバとすることができる。サーバ２０２－２は、音声データを蓄積しており、この蓄積は、上記したようにネット上で公開されている映像を解析し、物体と音声を取得することで行うことができる。よって、音声データを蓄積しているサーバとしては、既に構築されているサーバを用いても、本技術を適用した処理を行うことができる。 With such a configuration, the server 202-1 and the server 202-3 can be unique API (Application Programming Interface) servers. Further, the server 202-2 can be a public API server. The server 202-2 stores audio data, and this storage can be performed by analyzing the video published on the net as described above and acquiring the object and the audio. Therefore, even if a server that has already been constructed is used as the server that stores voice data, processing to which the present technology is applied can be performed.
　サーバ２０２－１とサーバ２０２－３は、振動を提示するための処理を行うサーバであり、１つのサーバとしても良い。 The server 202-1 and the server 202-3 are servers that perform processing for presenting vibration, and may be one server.
　フィードバックは、情報処理装置２０１から、サーバ２０２－１に供給される。サーバ２０２－１は、フィードバックが供給された場合、フィードバックの内容と、フィードバックの元になった認識物体情報（物体名など）を、サーバ２０２－２を介して、または直接的にサーバ２０２－３に供給する。 Feedback is supplied from the information processing device 201 to the server 202-1. When the feedback is supplied, the server 202-1 transmits the content of the feedback and the perceptible object information (object name, etc.) that is the source of the feedback via the server 202-2 or directly to the server 202-3. Supply to.
　サーバ２０２－３は、フィードバックと、フィードバックの元になった認識物体情報、およびフィードバックの基になった音声データを用いて、音声データから振動データに変換するときの変換方式を学習する。例えば、強化学習を用いて、認識物体情報を入力情報とし、振動データを出力情報としたときに、その振動データに対する報酬をフィードバックとした学習が行われる。 The server 202-3 learns a conversion method when converting from voice data to vibration data by using the feedback, the perceptible object information that is the source of the feedback, and the voice data that is the basis of the feedback. For example, using reinforcement learning, when the recognized object information is used as input information and the vibration data is used as output information, learning is performed using the reward for the vibration data as feedback.
　このように、情報処理装置２０１と、複数のサーバ２０２で処理が行われる構成とすることもできる。 In this way, the information processing device 201 and the plurality of servers 202 can be configured to perform processing.
　＜複数台の情報処理装置による動作例＞
　複数台の情報処理装置１０または情報処理装置２０１（以下情報処理装置１０を例に挙げて説明を続ける）で、連動して動作する場合について説明する。 <Example of operation using multiple information processing devices>
A case will be described in which a plurality of information processing devices 10 or information processing devices 201 (hereinafter, the information processing device 10 will be taken as an example to continue the description) will operate in conjunction with each other.
　例えば、図１５に示すように、情報処理装置１０－１と情報処理装置１０－２が、同一の物体（ここでは、ボール３０１とする）を撮像している。ボール３０１が情報処理装置１０で撮像されることで、ボール３０１に関する振動が、情報処理装置１０－１と情報処理装置１０－２において提示される。ボール３０１の振動データには、例えば、“他の端末が動かされたとき、振動を提示する”とのアクションが関連付けられていたとする。 For example, as shown in FIG. 15, the information processing device 10-1 and the information processing device 10-2 image the same object (here, the ball 301). When the ball 301 is imaged by the information processing device 10, the vibration related to the ball 301 is presented by the information processing device 10-1 and the information processing device 10-2. It is assumed that the vibration data of the ball 301 is associated with an action of "presenting vibration when another terminal is moved", for example.
　情報処理装置１０－１と情報処理装置１０－２が、図１０に示したフローチャートに基づく処理を実行する場合、ステップＳ１７においてユーザからのアクションが必要な物体であるか否かの判定を行うとき、上記したアクションが設定されているため、ＹＥＳと判定され、ステップＳ１８に処理は進められる。 When the information processing device 10-1 and the information processing device 10-2 execute the process based on the flowchart shown in FIG. 10, when determining in step S17 whether or not the object requires an action from the user. Since the above-mentioned action is set, it is determined as YES, and the process proceeds to step S18.
　ステップＳ１８において、ユーザからのアクションがあったか否かが判定されるが、この場合、“他の端末が動かされたとき、振動を提示する”というアクションであるため、他の端末（相手の状態）を監視し、相手の状態に変化があったときに、振動の提示が開始される。 In step S18, it is determined whether or not there is an action from the user. In this case, since it is an action of "presenting vibration when another terminal is moved", the other terminal (state of the other party). When there is a change in the other party's condition, the vibration presentation is started.
　例えば、情報処理装置１０－１が動かされた場合、情報処理装置１０－１は、動いたという情報を、情報処理装置１０－２に送信する。情報処理装置１０－２は、情報処理装置１０－１から、動いたという情報を受信した場合、その情報に対応する振動の提示を開始する。 For example, when the information processing device 10-1 is moved, the information processing device 10-1 transmits the information that the information processing device 10-1 has moved to the information processing device 10-2. When the information processing device 10-2 receives the information that it has moved from the information processing device 10-1, the information processing device 10-2 starts to present the vibration corresponding to the information.
　例えば、ボール３０１が撮像されている状態で、情報処理装置１０－１が振られた場合、情報処理装置１０－１から情報処理装置１０－２にボールが投げられた状況であると仮想し、情報処理装置１０－２では、ボールをキャッチしたときの擬似的な振動が提示される。 For example, when the information processing device 10-1 is shaken while the ball 301 is being imaged, it is assumed that the ball is thrown from the information processing device 10-1 to the information processing device 10-2. In the information processing device 10-2, a pseudo vibration when the ball is caught is presented.
　ユーザに提示される振動は、モーション、動きの速さ、距離、画面の数値、時間、相手の状況などに応じた振動とすることができる。 The vibration presented to the user can be a vibration according to the motion, the speed of movement, the distance, the numerical value on the screen, the time, the situation of the other party, and the like.
　このように、同一の物体を撮像しているときには、同一の物体を撮像している端末同士で、連動して動作するようにすることもできる。換言すれば、一方の動作により、他方の端末において振動が開始されるようにしたり、両方の端末で同じ振動が提示されるようにしたりすることができる。 In this way, when the same object is being imaged, the terminals that are imaging the same object can be made to operate in conjunction with each other. In other words, one action can cause the other terminal to initiate vibration, or both terminals to present the same vibration.
　また、そのような連動した動作を行わせるには、アクションを規定しておくことで可能となる。また、アクションを複数設定しておくことも可能であり、複数のアクションを設定しておくことで、様々な状況に応じた振動をユーザに提示することができるようになる。 Also, in order to perform such linked operations, it is possible by prescribing actions. It is also possible to set a plurality of actions, and by setting a plurality of actions, it becomes possible to present vibrations to the user according to various situations.
　情報処理システム２００に適用し、サーバ２０２が、複数の情報処理装置２０１に対する処理を実行するようにしても良い。サーバ２０２で蓄積されているデータを更新することで、様々な振動をユーザに提示することができる。 It may be applied to the information information system 200 so that the server 202 executes processing for a plurality of information processing devices 201. By updating the data stored in the server 202, various vibrations can be presented to the user.
　＜記録媒体について＞
　上述した一連の処理は、ハードウエアにより実行することもできるし、ソフトウエアにより実行することもできる。一連の処理をソフトウエアにより実行する場合には、そのソフトウエアを構成するプログラムが、コンピュータにインストールされる。ここで、コンピュータには、専用のハードウエアに組み込まれているコンピュータや、各種のプログラムをインストールすることで、各種の機能を実行することが可能な、例えば汎用のパーソナルコンピュータなどが含まれる。 <About recording media>
The series of processes described above can be executed by hardware or by software. When a series of processes are executed by software, the programs constituting the software are installed on the computer. Here, the computer includes a computer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.
　図１６は、上述した一連の処理をプログラムにより実行するコンピュータのハードウエアの構成例を示すブロック図である。コンピュータにおいて、ＣＰＵ（Central Processing Unit）１００１、ＲＯＭ（Read Only Memory）１００２、ＲＡＭ（Random Access Memory）１００３は、バス１００４により相互に接続されている。バス１００４には、さらに、入出力インタフェース１００５が接続されている。入出力インタフェース１００５には、入力部１００６、出力部１００７、記憶部１００８、通信部１００９、およびドライブ１０１０が接続されている。 FIG. 16 is a block diagram showing a configuration example of computer hardware that executes the above-mentioned series of processes programmatically. In a computer, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are connected to each other by a bus 1004. An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.
　入力部１００６は、キーボード、マウス、マイクロフォンなどよりなる。出力部１００７は、ディスプレイ、スピーカなどよりなる。記憶部１００８は、ハードディスクや不揮発性のメモリなどよりなる。通信部１００９は、ネットワークインタフェースなどよりなる。ドライブ１０１０は、磁気ディスク、光ディスク、光磁気ディスク、または半導体メモリなどのリムーバブルメディア１０１１を駆動する。 The input unit 1006 includes a keyboard, a mouse, a microphone, and the like. The output unit 1007 includes a display, a speaker, and the like. The storage unit 1008 includes a hard disk, a non-volatile memory, and the like. The communication unit 1009 includes a network interface and the like. The drive 1010 drives a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
　以上のように構成されるコンピュータでは、ＣＰＵ１００１が、例えば、記憶部１００８に記憶されているプログラムを、入出力インタフェース１００５およびバス１００４を介して、ＲＡＭ１００３にロードして実行することにより、上述した一連の処理が行われる。 In the computer configured as described above, the CPU 1001 loads the program stored in the storage unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes the above-described series. Is processed.
　コンピュータ（ＣＰＵ１００１）が実行するプログラムは、例えば、パッケージメディア等としてのリムーバブルメディア１０１１に記録して提供することができる。また、プログラムは、ローカルエリアネットワーク、インターネット、デジタル衛星放送といった、有線または無線の伝送媒体を介して提供することができる。 The program executed by the computer (CPU1001) can be recorded and provided on the removable media 1011 as a package media or the like, for example. Programs can also be provided via wired or wireless transmission media such as local area networks, the Internet, and digital satellite broadcasting.
　コンピュータでは、プログラムは、リムーバブルメディア１０１１をドライブ１０１０に装着することにより、入出力インタフェース１００５を介して、記憶部１００８にインストールすることができる。また、プログラムは、有線または無線の伝送媒体を介して、通信部１００９で受信し、記憶部１００８にインストールすることができる。その他、プログラムは、ＲＯＭ１００２や記憶部１００８に、予めインストールしておくことができる。 In the computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by mounting the removable media 1011 in the drive 1010. Further, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. In addition, the program can be pre-installed in the ROM 1002 or the storage unit 1008.
　なお、コンピュータが実行するプログラムは、本明細書で説明する順序に沿って時系列に処理が行われるプログラムであっても良いし、並列に、あるいは呼び出しが行われたとき等の必要なタイミングで処理が行われるプログラムであっても良い。 The program executed by the computer may be a program that is processed in chronological order in the order described in this specification, or may be a program that is processed in parallel or at a necessary timing such as when a call is made. It may be a program in which processing is performed.
　また、本明細書において、システムとは、複数の装置により構成される装置全体を表すものである。 Further, in the present specification, the system represents the entire device composed of a plurality of devices.
　なお、本明細書に記載された効果はあくまで例示であって限定されるものでは無く、また他の効果があってもよい。 Note that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.
　なお、本技術の実施の形態は、上述した実施の形態に限定されるものではなく、本技術の要旨を逸脱しない範囲において種々の変更が可能である。 It should be noted that the embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.
　なお、本技術は以下のような構成も取ることができる。
（１）
　撮影された画像内の物体を認識する認識部と、
　前記認識部により認識された前記物体の状態に応じた振動データを生成する振動データ生成部と
　を備える情報処理装置。
（２）
　前記物体の状態は、基準画像との差分であり、
　前記振動データ生成部は、前記差分に応じた補正を行う
　前記（１）に記載の情報処理装置。
（３）
　前記物体の状態は、前記認識部により認識された前記物体の個数または量と、予め設定されている物体の個数または量の差分であり、
　前記振動データ生成部は、前記差分に応じた補正を行う
　前記（１）または（２）に記載の情報処理装置。
（４）
　前記振動データ生成部は、前記基準画像と関連付けられている振動データを、前記差分に応じて補正する
　前記（２）に記載の情報処理装置。
（５）
　前記振動データ生成部は、前記物体の個数または量と関連付けられている振動データを、前記差分に応じて補正する
　前記（３）に記載の情報処理装置。
（６）
　前記振動データ生成部は、
　前記基準画像と関連付けられている音声データを取得し、
　前記音声データを、前記差分に応じて補正し、
　補正された前記音声データを、振動データに変換する
　前記（２）に記載の情報処理装置。
（７）
　前記振動データ生成部は、
　前記物体の個数または量と関連付けられている音声データを取得し、
　前記音声データを、前記差分に応じて補正し、
　補正された前記音声データを、振動データに変換する
　前記（３）に記載の情報処理装置。
（８）
　ユーザからのフィードバックにより、前記振動データを修正する
　前記（１）に記載の情報処理装置。
（９）
　ユーザからのフィードバックにより、前記音声データを修正する
　前記（６）に記載の情報処理装置。
（１０）
　前記振動データに基づく振動は、ユーザのアクションがあったときに提示する
　前記（１）乃至（９）のいずれかに記載の情報処理装置。
（１１）
　情報処理装置が、
　撮影された画像内の物体を認識し、
　認識された前記物体の状態に応じた振動データを生成する
　情報処理方法。
（１２）
　コンピュータに、
　撮影された画像内の物体を認識し、
　認識された前記物体の状態に応じた振動データを生成する
　ステップを含む処理を実行させるためのプログラム。 The present technology can also have the following configurations.
(1)
A recognition unit that recognizes objects in the captured image,
An information processing device including a vibration data generation unit that generates vibration data according to the state of the object recognized by the recognition unit.
(2)
The state of the object is a difference from the reference image.
The information processing device according to (1) above, wherein the vibration data generation unit corrects according to the difference.
(3)
The state of the object is a difference between the number or amount of the object recognized by the recognition unit and the preset number or amount of the object.
The information processing apparatus according to (1) or (2), wherein the vibration data generation unit corrects according to the difference.
(4)
The information processing device according to (2), wherein the vibration data generation unit corrects vibration data associated with the reference image according to the difference.
(5)
The information processing device according to (3), wherein the vibration data generation unit corrects vibration data associated with the number or amount of the objects according to the difference.
(6)
The vibration data generator
Acquire the audio data associated with the reference image and
The voice data is corrected according to the difference,
The information processing device according to (2) above, which converts the corrected voice data into vibration data.
(7)
The vibration data generator
Acquires audio data associated with the number or quantity of the objects and
The voice data is corrected according to the difference,
The information processing device according to (3) above, which converts the corrected voice data into vibration data.
(8)
The information processing device according to (1) above, wherein the vibration data is corrected by feedback from the user.
(9)
The information processing device according to (6) above, wherein the voice data is modified by feedback from the user.
(10)
The information processing device according to any one of (1) to (9) above, wherein the vibration based on the vibration data is presented when there is an action of the user.
(11)
The information processing device
Recognize objects in the captured image and
An information processing method that generates vibration data according to the recognized state of the object.
(12)
On the computer
Recognize objects in the captured image and
A program for executing a process including a step of generating vibration data according to the recognized state of the object.
　１０　情報処理装置，　１１　ＣＰＵ，　１２　ＲＯＭ，　１３　ＲＡＭ，　１４　センサ部，　１５　入出力インタフェース，　１６　表示部，　１７　タッチ入力部，　１８　通信部，　１９　ホームボタン，　２０　振動子，　４１　撮像部，　４２　物体認識部，　４３　振動データ取得部，　４４　振動データ記憶部，　４５　差分判定部，　４７　属性パラメータ記憶部，　４８　補正処理部，　４９　振動生成提示部，　２００　情報処理システム，　２０１　情報処理装置，　２０２　サーバ，　２１１　撮像部，　２１２　物体認識部，　２１３　通信部，　２１４　振動生成提示部，　２２１　通信部，　２２２　音声データ取得部，　２２３　音声データ記憶部，　２２４　差分判定部，　２２５　属性パラメータ取得部，　２２６　属性パラメータ記憶部，　２２７　振動データ生成部 10 information processing device, 11 CPU, 12 ROM, 13 RAM, 14 sensor unit, 15 input / output interface, 16 display unit, 17 touch input unit, 18 communication unit, 19 home button, 20 oscillator, 41 imaging unit, 42 objects Recognition unit, 43 vibration data acquisition unit, 44 vibration data storage unit, 45 difference judgment unit, 47 attribute parameter storage unit, 48 correction processing unit, 49 vibration generation presentation unit, 200 information processing system, 201 information processing device, 202 server, 211 Imaging unit, 212 Object recognition unit, 213 Communication unit, 214 Vibration generation presentation unit, 221 Communication unit, 222 Audio data acquisition unit, 223 Audio data storage unit, 224 Difference judgment unit, 225 Attribute parameter acquisition unit, 226 Attribute parameter storage Department, 227 Vibration data generation unit

Claims (12)

  1. 　撮影された画像内の物体を認識する認識部と、
    　前記認識部により認識された前記物体の状態に応じた振動データを生成する振動データ生成部と
    　を備える情報処理装置。     A recognition unit that recognizes objects in the captured image,
    An information processing device including a vibration data generation unit that generates vibration data according to the state of the object recognized by the recognition unit.
  2. 　前記物体の状態は、基準画像との差分であり、
    　前記振動データ生成部は、前記差分に応じた補正を行う
    　請求項１に記載の情報処理装置。     The state of the object is a difference from the reference image.
    The information processing device according to claim 1, wherein the vibration data generation unit corrects according to the difference.
  3. 　前記物体の状態は、前記認識部により認識された前記物体の個数または量と、予め設定されている物体の個数または量の差分であり、
    　前記振動データ生成部は、前記差分に応じた補正を行う
    　請求項１に記載の情報処理装置。     The state of the object is a difference between the number or amount of the object recognized by the recognition unit and the preset number or amount of the object.
    The information processing device according to claim 1, wherein the vibration data generation unit corrects according to the difference.
  4. 　前記振動データ生成部は、前記基準画像と関連付けられている振動データを、前記差分に応じて補正する
    　請求項２に記載の情報処理装置。     The information processing device according to claim 2, wherein the vibration data generation unit corrects vibration data associated with the reference image according to the difference.
  5. 　前記振動データ生成部は、前記物体の個数または量と関連付けられている振動データを、前記差分に応じて補正する
    　請求項３に記載の情報処理装置。     The information processing device according to claim 3, wherein the vibration data generation unit corrects vibration data associated with the number or amount of the objects according to the difference.
  6. 　前記振動データ生成部は、
    　前記基準画像と関連付けられている音声データを取得し、
    　前記音声データを、前記差分に応じて補正し、
    　補正された前記音声データを、振動データに変換する
    　請求項２に記載の情報処理装置。     The vibration data generator
    Acquire the audio data associated with the reference image and
    The voice data is corrected according to the difference,
    The information processing apparatus according to claim 2, wherein the corrected voice data is converted into vibration data.
  7. 　前記振動データ生成部は、
    　前記物体の個数または量と関連付けられている音声データを取得し、
    　前記音声データを、前記差分に応じて補正し、
    　補正された前記音声データを、振動データに変換する
    　請求項３に記載の情報処理装置。     The vibration data generator
    Acquires audio data associated with the number or quantity of the objects and
    The voice data is corrected according to the difference,
    The information processing apparatus according to claim 3, wherein the corrected voice data is converted into vibration data.
  8. 　ユーザからのフィードバックにより、前記振動データを修正する
    　請求項１に記載の情報処理装置。     The information processing apparatus according to claim 1, wherein the vibration data is corrected by feedback from the user.
  9. 　ユーザからのフィードバックにより、前記音声データを修正する
    　請求項６に記載の情報処理装置。     The information processing apparatus according to claim 6, wherein the voice data is modified by feedback from the user.
  10. 　前記振動データに基づく振動は、ユーザのアクションがあったときに提示する
    　請求項１に記載の情報処理装置。     The information processing device according to claim 1, wherein the vibration based on the vibration data is presented when there is an action of the user.
  11. 　情報処理装置が、
    　撮影された画像内の物体を認識し、
    　認識された前記物体の状態に応じた振動データを生成する
    　情報処理方法。     The information processing device
    Recognize objects in the captured image and
    An information processing method that generates vibration data according to the recognized state of the object.
  12. 　コンピュータに、
    　撮影された画像内の物体を認識し、
    　認識された前記物体の状態に応じた振動データを生成する
    　ステップを含む処理を実行させるためのプログラム。     On the computer
    Recognize objects in the captured image and
    A program for executing a process including a step of generating vibration data according to the recognized state of the object.
PCT/JP2020/027777 2019-08-02 2020-07-17 Information processing device, information processing method, and program WO2021024753A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019-143032 2019-08-02
JP2019143032 2019-08-02

Publications (1)

Publication Number Publication Date
WO2021024753A1 true WO2021024753A1 (en) 2021-02-11

Family

ID=74504062

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/027777 WO2021024753A1 (en) 2019-08-02 2020-07-17 Information processing device, information processing method, and program

Country Status (1)

Country Link
WO (1) WO2021024753A1 (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013168732A1 (en) * 2012-05-08 2013-11-14 株式会社ニコン Electronic device
JP2017117370A (en) * 2015-12-25 2017-06-29 富士通テン株式会社 Input device and control method of input device
JP2017518691A (en) * 2014-05-13 2017-07-06 クアルコム，インコーポレイテッド System and method for providing haptic feedback to assist in image capture

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013168732A1 (en) * 2012-05-08 2013-11-14 株式会社ニコン Electronic device
JP2017518691A (en) * 2014-05-13 2017-07-06 クアルコム，インコーポレイテッド System and method for providing haptic feedback to assist in image capture
JP2017117370A (en) * 2015-12-25 2017-06-29 富士通テン株式会社 Input device and control method of input device

Similar Documents

Publication Publication Date Title
WO2016185809A1 (en) Information processing apparatus, information processing method, and program
CN109379613B (en) Audio and video synchronization adjustment method, television, computer readable storage medium and system
WO2013168732A1 (en) Electronic device
US20130041976A1 (en) Context-aware delivery of content
JP6300792B2 (en) Enhancing captured data
WO2020108102A1 (en) Vibration method, electronic device and storage medium
US20200076866A1 (en) Systems, devices, and methods for streaming haptic effects
RU2665270C2 (en) Method and device for information processing
US11331569B2 (en) Information processing device, information processing method, program, and information storage medium
CN105828166A (en) Recording method of terminal screen display content and terminal
US20190163438A1 (en) Information processing apparatus and information processing method
JP2009171176A (en) Reproduction apparatus, its control method, and program
JP2012043046A (en) Conference minute creation system and program
US10468018B2 (en) Methods and systems for recognizing audio played and recording related video for viewing
CN106782459B (en) Piano automatic playing control system and method based on mobile terminal application program
WO2021024753A1 (en) Information processing device, information processing method, and program
JP5729844B1 (en) Content evaluation apparatus, system, server apparatus, and terminal apparatus
CN114121028A (en) Voice playing method, device, equipment and storage medium
US20120254751A1 (en) Apparatus and method for processing sound source
JP2006119320A (en) Electronic music device system, server side electronic music device, and client side electronic music device
JP7424468B2 (en) Parameter inference method, parameter inference system, and parameter inference program
CN110120215B (en) Audio processing method, device and system
CN113241057A (en) Interactive method, apparatus, system and medium for speech synthesis model training
CN114710553A (en) Information acquisition method, information push method and terminal equipment
WO2023166721A1 (en) Information processing apparatus, information processing system, and information processing method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20850550

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20850550

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP