WO2022102307A1 - 信号処理装置、信号処理方法、信号処理システム - Google Patents
信号処理装置、信号処理方法、信号処理システム Download PDFInfo
- Publication number
- WO2022102307A1 WO2022102307A1 PCT/JP2021/037381 JP2021037381W WO2022102307A1 WO 2022102307 A1 WO2022102307 A1 WO 2022102307A1 JP 2021037381 W JP2021037381 W JP 2021037381W WO 2022102307 A1 WO2022102307 A1 WO 2022102307A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- tactile
- user
- tactile signal
- signal
- adjustment
- Prior art date
Links
- 238000012545 processing Methods 0.000 title claims abstract description 170
- 238000003672 processing method Methods 0.000 title claims description 5
- 238000000034 method Methods 0.000 claims description 105
- 230000008569 process Effects 0.000 claims description 92
- 238000004891 communication Methods 0.000 claims description 56
- 230000005540 biological transmission Effects 0.000 claims description 31
- 230000035807 sensation Effects 0.000 claims description 14
- 230000000007 visual effect Effects 0.000 claims description 14
- 230000008859 change Effects 0.000 claims description 10
- 238000005516 engineering process Methods 0.000 abstract description 9
- 230000036544 posture Effects 0.000 description 101
- 210000003127 knee Anatomy 0.000 description 13
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 11
- 125000002066 L-histidyl group Chemical group [H]N1C([H])=NC(C([H])([H])[C@](C(=O)[*])([H])N([H])[H])=C1[H] 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 230000010365 information processing Effects 0.000 description 5
- 230000004936 stimulating effect Effects 0.000 description 5
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 210000001991 scapula Anatomy 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 230000003187 abdominal effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000008054 signal transmission Effects 0.000 description 2
- 230000000638 stimulation Effects 0.000 description 2
- 210000001015 abdomen Anatomy 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000007664 blowing Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001151 other effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/25—Output arrangements for video game devices
- A63F13/28—Output arrangements for video game devices responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/23—Recognition of whole body movements, e.g. for sport training
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/4223—Cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/013—Force feedback applied to a game
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/454—Content or additional data filtering, e.g. blocking advertisements
- H04N21/4542—Blocking scenes or portions of the received content, e.g. censoring scenes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/466—Learning process for intelligent management, e.g. learning user preferences for recommending movies
- H04N21/4668—Learning process for intelligent management, e.g. learning user preferences for recommending movies for recommending content, e.g. movies
Definitions
- This technique relates to a signal processing device, a signal processing method, and a signal processing system, and particularly to a technique related to a signal processing device that generates a tactile signal for tactile presentation.
- the creator of the tactile signal generates a tactile signal for providing an appropriate tactile stimulus in consideration of the content and the like, and the adjustment of the tactile signal appropriately reflects the creator's intention. It is desirable to be done.
- the user experienced person adjusts after experiencing and confirming the tactile stimulus by himself / herself, and the adjustment reflects the intention of the creator. Not necessarily.
- This technique was made in view of such circumstances, and aims to adjust the tactile signal so that an appropriate tactile stimulus that reflects the creator's intention is provided to the user.
- the signal processing device has an information acquisition unit that acquires state information indicating the user's state including the posture of the user, and a tactile signal acquisition unit that acquires a reference tactile signal for presenting a tactile sensation to a user in a reference state. And an adjustment processing unit that generates an adjustment tactile signal by adjusting the reference tactile signal based on the state information. Different postures of the user also result in different tactile stimuli experienced by the user. According to this configuration, by adjusting the reference tactile signal based on the state information indicating the user's state including the posture, the adjusted tactile signal according to the user's state is generated.
- the signal processing device described above may include a determination processing unit for determining whether or not the adjustment can be executed.
- Appropriate tactile stimuli may or may not differ depending on the posture of the user. According to this configuration, it is possible to appropriately switch between the case where the adjustment is performed based on the user's state information and the case where the adjustment is not performed.
- the determination processing unit in the above-mentioned signal processing device may determine the determination result of whether or not the execution is possible based on the operation of the user. Whether or not to adjust the tactile signal may be determined not only automatically according to the type of the tactile signal but also by the user.
- the above-mentioned signal processing device may include an identification information setting unit that sets identification information indicating whether or not the adjustment is to be made in the reference tactile signal.
- the adjustment processing unit can determine whether or not to perform adjustment by confirming the identification information for each reference tactile signal.
- the determination processing unit in the above-mentioned signal processing device may determine whether or not the adjustment is necessary because the posture of the user and the reference state are different. This makes it possible to appropriately determine whether or not the reference tactile signal should be adjusted, taking into account the posture of the user.
- the determination processing unit in the signal processing apparatus described above may determine whether or not the adjustment is impossible based on the state information and the reference state. For example, if the user's state (posture, etc.) is too far from the reference state, it may not be possible for the user to experience an appropriate tactile stimulus even if the reference tactile signal is adjusted.
- the above-mentioned signal processing device includes a communication control unit that controls transmission of a tactile signal to a tactile signal reproduction device that reproduces the tactile signal, and the communication control unit is determined to be unable to make the adjustment.
- transmission control may be performed so that the tactile signal reproduction is not performed by the tactile signal reproduction device.
- the communication control unit in the above-mentioned signal processing device may not transmit a tactile signal when it is determined that the adjustment is impossible. This reduces the processing load of the tactile signal transmission process.
- the communication control unit in the above-mentioned signal processing device may transmit the tactile signal by adding control information so that the tactile signal is not reproduced when it is determined that the adjustment is impossible.
- the signal processing device that adjusts the tactile signal can uniformly transmit the tactile signal regardless of the necessity of adjustment. For example, when transmitting a tactile signal to a plurality of tactile signal reproducing devices, it is not necessary to control so that the tactile signal is not transmitted only to a specific tactile signal worn by a user who takes a posture significantly different from the reference state.
- the above-mentioned signal processing device includes a communication control unit that controls transmission to transmit a tactile signal to a tactile signal reproduction device that reproduces a tactile signal, and the information acquisition unit visually recognizes an image to be visually recognized by the user.
- the communication control unit acquires the visual information indicating whether or not the image is visible, and when the information indicating that the user is not visually recognizing the image is acquired as the visual information, the reproduction of the tactile signal is the tactile signal.
- Transmission control may be performed so that it is not performed by the playback device.
- the user may experience the tactile stimulus in various situations. And, in some cases, it may or may not be concentrated on the content. For example, as a method for determining whether or not a user who enjoys a tactile stimulus together with an image is concentrating on the content, it is conceivable to determine whether or not the image is properly viewed.
- the above-mentioned signal processing device includes a communication control unit that controls transmission of a tactile signal to a tactile signal reproduction device that reproduces the tactile signal, and the communication control unit adjusts when transmitting the adjusted tactile signal. Transmission control may be performed to notify that the above has been performed. As a result, the user can know that the reference tactile signal is adjusted.
- the adjustment processing unit in the signal processing device described above may make adjustments to change the intensity of the tactile presentation in the adjustment of the reference tactile signal. This makes it possible to present a tactile sensation of appropriate strength according to the state of the user.
- the adjustment processing unit in the above-mentioned signal processing device may perform the adjustment according to the tactile signal reproduction device worn by the user. This makes it possible to transmit a tactile signal that has been appropriately adjusted according to not only the difference in the user's state but also the difference in the tactile signal reproducing device worn.
- the adjustment processing unit in the above-mentioned signal processing device may adjust the intensity and timing of the tactile presentation according to the position of the tactile presentation unit included in the tactile signal reproduction device and the state information. As a result, even if each user wears a tactile signal reproducing device having a different arrangement position of the vibrating portion, an appropriate tactile signal is generated for each user. Further, the frequency characteristic of the tactile presentation signal may be adjusted according to the reproduction frequency characteristic of the tactile presentation unit included in the tactile signal reproduction device.
- the information acquisition unit in the above-mentioned signal processing device acquires distance information between the display device and the user on which an image to be visually recognized by the user is displayed, and the adjustment processing unit presents the tactile sensation according to the acquired distance information. Adjustments may be made to change the strength of the. When the user enjoys the tactile stimulus while watching the image, if the appearance of the image is different, the optimum tactile stimulus is likely to be different.
- the information acquisition unit in the signal processing device may acquire the state information each time the posture of the user changes. As the user's posture changes, so does the optimal tactile stimulus.
- the signal processing method executed by the signal processing device acquires a process of acquiring state information indicating the user's state including the posture of the user and a reference tactile signal for presenting a tactile sensation to the user in the reference state. It includes a process and a process of generating an adjusted tactile signal by adjusting the reference tactile signal based on the state information.
- the signal processing system has an information acquisition unit that acquires state information indicating the user's state including the posture of the user, and a tactile signal acquisition unit that acquires a reference tactile signal for presenting a tactile sensation to a user in a reference state.
- a signal processing device including an adjustment processing unit that generates an adjustment tactile signal by adjusting the reference tactile signal based on the state information, a tactile signal reproduction device that reproduces the adjustment tactile signal, and the like. It is equipped with.
- the implementation of the present technique is not limited to this.
- it may provide a new experience to the user by combining acoustic data such as music and voice with tactile stimuli.
- the game may be made more enjoyable by providing the tactile stimulus according to the video data and the acoustic data included in the game content.
- information other than video data and acoustic data may be provided to the user in combination with tactile stimuli.
- the entertainment system 1 includes a cinema system 2 that allows a user to view video and audio in movie content, and a providing system 3 as a signal processing system that performs signal processing for providing a tactile stimulus based on a tactile signal to the user. There is.
- the cinema system 2 includes a playback control device 21, a projector 22, an acoustic output device 23, and a screen 24.
- the reproduction control device 21 outputs the video data of the movie content to the projector 22 and outputs the acoustic data to the acoustic output device 23. Further, the reproduction control device 21 transmits a time code indicating the reproduction position of the video content to the provision system 3 in order to synchronize the cinema system 2 with the provision system 3 described later.
- the projector 22 displays an image on the screen 24 by performing a projection process on the image data of the movie content based on the instruction from the reproduction control device 21.
- the acoustic output device 23 performs acoustic output synchronized with the video content based on the instruction from the reproduction control device 21.
- the provided system 3 includes a presentation control device 31, a best-type tactile signal reproduction device 32, and a tactile DB (Database) 33.
- the tactile signal reproduction device 32 is a tactile presentation device, and may be one or a plurality of tactile signal reproduction devices 32. Further, one user may wear (or use) one tactile signal reproducing device 32, or one user may wear / use a plurality of tactile signal reproducing devices 32. For example, one user may wear the best-type tactile signal reproduction device 32 and the tactile signal reproduction device 32 to be attached to the foot or arm.
- the presentation control device 31 receives the time code from the cinema system 2 and controls the tactile signal reproduction device 32 based on the time code. Specifically, the presentation control device 31 transmits a tactile signal to the tactile signal reproducing device 32 at an appropriate timing according to the time code.
- the tactile signal to be transmitted includes a “reference tactile signal” which is an unadjusted signal and a “adjusted tactile signal” which is an adjusted reference tactile signal.
- the tactile signal reproduction device 32 may be a device such as a seat type (chair type) that is used without being worn by the user, in addition to the device that can be worn by the user.
- the presentation control device 31 manages the tactile DB 33 in which the time code, the tactile signal to be reproduced, and various control information are associated and stored.
- the presentation control device 31 acquires a reference tactile signal to be presented from the tactile DB 33 according to the received time code, and transmits the reference tactile signal as it is or the adjusted tactile signal adjusted to the tactile signal reproduction device 32.
- the presentation control device 31 includes an information acquisition unit 41, a tactile signal acquisition unit 42, a determination processing unit 43, an identification information setting unit 44, an adjustment processing unit 45, and a communication control unit 46. (See FIG. 2).
- the tactile signal reproduction device 32 is a device that generates vibration or the like based on the tactile signal created by the creator, and is, for example, in the shape of a vest type that can be worn by the user or a chair type that can be used. .. In the following description, the best type tactile signal reproduction device 32 will be given as a main example.
- the tactile signal reproduction device 32 includes a reproduction control unit 51, a vibration unit 52, a posture detection unit 53 that detects the posture of the wearing user, and a communication control unit 54 that receives a tactile signal via wireless communication. (See Fig. 3).
- the information acquisition unit 41 of the presentation control device 31 includes "state information” indicating the user's state including the posture of the user, "visual information” indicating whether or not the user is watching an image, and the user and the screen 24. It performs a process of acquiring "distance information” indicating a distance, "device information” about a tactile signal reproducing device 32, and the like.
- the state information, visual information, distance information, device information, etc. are used to determine whether or not to adjust the reference tactile signal in order to provide an appropriate tactile stimulus to the user.
- the state information is information indicating the user's state such as the user's posture information.
- the state information is different depending on the posture of sitting on a chair (FIG. 4), the posture of standing (FIG. 5), the posture of lying down (FIG. 6), and the like.
- the state information differs not only depending on the posture but also on the orientation of the user's face.
- the state information is different from the posture of viewing the image transferred to the screen 24. That is, the state information also includes information that can identify whether the user is sitting facing the front or facing sideways.
- the state information differs not only depending on the posture and the orientation of the face, but also on the relationship between the user and the chair or sofa. For example, as shown in FIG. 8, a user sitting on a chair with his / her back on the backrest and a user sitting on a chair without his / her back on the backrest as shown in FIG. 9 have state information. It will be different.
- the information acquisition unit 41 may acquire the state information periodically, may acquire it every time the user's state changes, or once at the time when the content reproduction starts regardless of the change in the user's state. You may only get it.
- the user's status information is acquired periodically, it is not necessary to execute a process of determining whether or not to perform transmission in the device that transmits the status information, so that the processing load can be reduced.
- the necessary information is acquired at the required timing, so that the amount of communication is reduced and the processing load related to communication is reduced. be able to.
- the state information is acquired only once regardless of the change in the posture of the user, the communication amount and the processing load related to the transmission and reception of the state information can be significantly reduced.
- the user's state information may be acquired from the tactile signal reproduction device 32 or may be acquired from other devices.
- the tactile signal reproduction device 32 includes an IMU (Inertial Measurement Unit) and a pressure sensitive element, and acquires state information about the user based on the information acquired from the IMU (Inertial Measurement Unit) and the pressure sensitive element.
- the IMU data may be acquired from the tactile signal reproduction device 32, and the information acquisition unit 41 may acquire the state information by estimating the posture of the user or the like based on the received IMU data.
- the state information may be acquired from the tactile signal reproduction device 32 by executing the process of estimating the user's posture from the IMU data in the tactile signal reproduction device 32.
- the information of the tactile signal reproduction device 32 As the device information, as the information of the tactile signal reproduction device 32, information such as the number of the vibrating unit 52 included in the tactile signal reproduction device 32, the position of the vibrating unit 52, and the frequency characteristics of the vibrating unit 52 is acquired from the tactile signal reproduction device 32. .. These device information are used in the adjustment process of the reference tactile signal.
- the characteristic information for each type of device may be stored in the DB as the device information, and the information acquisition unit 41 may acquire the device information from the DB based on the model number information acquired from the tactile signal reproduction device 32.
- the reference tactile signal is generated on the assumption that it is reproduced by a specific tactile signal reproducing device 32. That is, when the tactile signal reproduction device 32 is different from the assumed tactile signal reproduction device 32, it may be necessary to adjust the reference tactile signal.
- the device information can also be said to be information for identifying the difference between the assumed specific tactile signal reproduction device 32 (reference device) and the tactile signal reproduction device 32 being used by the user. Then, when the tactile signal reproducing device 32 being used by the user is different from the reference device, the adjustment process described later is performed.
- the visual recognition information is an example of information used for determining whether or not the user is concentrating on viewing the content, and can be acquired based on, for example, image processing or IMU information.
- image for acquiring the viewing information various images such as an image captured by an image pickup device arranged in the vicinity of the screen 24 can be used.
- IMU data when the user is concentrating on the content and IMU data when the user is enjoying the content from time to time while doing other things are collected, and the current user can use machine learning or the like. It may be determined whether or not the content is concentrated and obtained as visual information.
- Distance information can be obtained by various methods.
- an image pickup device capable of outputting a distance image may be arranged in the vicinity of the screen 24, and distance information indicating the distance between the user and the screen 24 may be obtained by calculating the distance between the user as a subject and the image pickup device. ..
- the tactile signal acquisition unit 42 acquires the reference tactile signal before adjustment from the tactile DB 33.
- the determination processing unit 43 performs processing for determining whether or not to adjust the reference tactile signal. For example, it is determined that the adjustment process is executed when the posture of the user or the like is different from the reference state.
- the standing state (FIG. 5) is based on the state of sitting with the back attached to the backrest of the chair and looking at the screen 24 arranged in front (FIGS. 4 and 8). If you are lying down (Fig. 6), sitting sideways (Fig. 7), or your back is not on the backrest (Fig. 9), it is determined that the adjustment process is different from the standard state. do.
- IMU data when the user is in the reference state is stored as reference data, and the degree of deviation between the acquired IMU data and the reference data is determined. It may be done by judgment.
- the determination processing unit 43 may determine whether or not to adjust the reference tactile signal based on other information. For example, some tactile stimuli provided to the user may not require adjustment. The creator may specify that the tactile signal related to such a tactile stimulus is a tactile signal that cannot be adjusted or does not need to be adjusted. The determination processing unit 43 acquires flag information and the like as such designated information, and determines whether or not to adjust the reference tactile signal.
- the flag information indicating that the tactile signal cannot be adjusted or does not need to be adjusted may be set by the user.
- the determination processing unit 43 may determine that the user does not want to adjust the reference tactile signal based on the user's operation information or the like, and that the reference tactile signal is not adjusted.
- the identification information setting unit 44 adds the determination result by the determination processing unit 43 to the reference tactile signal in the form of flag information or the like.
- This flag information is used as identification information for the adjustment processing unit 45 to identify whether the reference tactile signal is a tactile signal to be adjusted or a tactile signal that does not require adjustment.
- Each reference tactile signal may be a series of reference tactile signals provided to the user in one scene in a movie, or may be each reference tactile signal included in one packet data. It should be noted that one packet data includes a tactile signal having a relatively short reproduction time, for example, a reproduction time of several msec to several hundred msec.
- the adjustment processing unit 45 performs adjustment processing on the reference tactile signal based on the identification information given to each reference tactile signal by the identification information setting unit 44, and generates an adjusted tactile signal.
- the adjusted tactile signal is, for example, a tactile signal adjusted to provide a comfortable tactile stimulus to a user who is in a state different from the reference state. Specifically, it will be described later.
- the communication control unit 46 performs information transmission / reception processing with the tactile signal reproduction device 32 via wireless or wired communication.
- the information to be transmitted to the tactile signal reproducing device 32 is a reference tactile signal, an adjusted tactile signal, or the like. Further, the information received from the tactile signal reproduction device 32 is user status information and the like.
- the reproduction control unit 51 of the tactile signal reproduction device 32 outputs a drive signal corresponding to the tactile signal received from the presentation control device 31 of the received tactile signal to each vibration unit 52 via wireless or wired communication.
- the tactile signal is assumed to include, for example, control information associated with a time code in order to synchronize with the reproduced content.
- the tactile signal includes information for specifying the time code and the vibrating unit 52 to be driven, information for specifying the operation pattern, information on the presentation intensity of the tactile signal, and the like.
- the vibrating unit 52 is configured to be vibrable by being provided with an actuator or the like, and can transmit a tactile stimulus to an operator by driving based on a tactile signal generated based on a tactile signal.
- FIG. 1 shows six vibrating portions 52 arranged on the front side. The number and position of the vibrating portions 52 may differ depending on the tactile signal reproducing device 32.
- the reproduction control unit 51 manages the time code transmitted from the cinema system 2.
- the communication control unit 54 performs wireless or wired communication with the communication control unit 46 of the presentation control device 31.
- the tactile signal reproduction device 32 can receive the reference tactile signal and the adjusted tactile signal from the presentation control device 31, and can transmit the user's state information including the posture information to the presentation control device 31. ing.
- Adjustment process The adjustment processing of the tactile signal executed by the adjustment processing unit 45 of the presentation control device 31 will be described with some examples.
- FIG. 10 is a diagram showing a state in which a scene in which the water surface rises from the feet of the characters in the movie is projected on the screen 24.
- processing for adjusting the reference tactile signal may be performed.
- the tactile stimulus provided to the user is created assuming that the user takes a general posture.
- the posture while sitting on the chair is regarded as the reference posture.
- the reference tactile signal is created assuming a state in which the body is turned diagonally upward by using the reclining function provided in the chair, that state is regarded as the reference posture.
- the adjustment process may be executed. For example, when the reference tactile signal is created on the assumption that the user has his / her back rested on the backrest of the chair, the state in which the back is attached to the backrest is regarded as the reference state. A user who does not have his / her back on the backrest even if the posture is the same as the reference posture may be determined not to be in the reference state assumed at the time of creating the reference tactile signal.
- the reference tactile signal is the tactile signal worn by the user as it is. It is transmitted to the playback device 32. If the position, number, and characteristics of the vibrating unit 52 included in the tactile signal reproducing device 32 are different from the assumptions, it may be necessary to adjust the reference tactile signal. When such an adjustment process is performed by the presentation control device 31, the adjusted tactile signal adjusted with the reference tactile signal is transmitted to the tactile signal reproducing device 32. Further, when the adjustment process is performed on the tactile signal reproduction device 32 side, the reference tactile signal is transmitted to the tactile signal reproduction device 32.
- the tactile stimulus given to the user as the reference state in the scene shown in FIG. 10 will be described.
- the user it is possible for the user to wear or use the tactile signal reproducing device 32 to give stimuli at eight points on the front side (ventral side) of the user. That is, it is possible to give tactile stimuli to the left and right shin parts P1R, P1L, the left and right knee parts P2R, P2L, the left and right waist parts P3R, P3L, and the left and right chest parts P4R, P4L, respectively (Fig.). 11).
- the user By stimulating each part of the user at the timing as shown in FIG. 13, the user can perceive the state where the water surface rises from the user's feet, and the sense of presence can be increased.
- the tactile stimulus given to the user who is in a standing state in the scene shown in FIG. 10 will be described.
- the shin portions P1R, P1L, Q1R, and Q1L are stimulated, and then the knee portions P2R, P2L, Q2R, and Q2L are stimulated after a predetermined time.
- the waist portion P3R, P3L, Q3R, Q3L is stimulated, and after the predetermined time elapses, the user's chest portion P4R, P4L and the scapula portion Q4R, Q4L are stimulated substantially at the same time.
- the user By stimulating each part of the user at the timing as shown in FIG. 14, that is, by stimulating the knee parts P2R, P2L, Q2R, Q2L and the waist parts P3R, P3L, Q3R, Q3L with a time lag, the user Can perceive a state in which the water surface rises from the feet while standing, and can increase the sense of presence.
- the tactile stimulus given to the user who is in a lying state in the scene shown in FIG. 10 will be described.
- stimulation to the shin portion Q1R, Q1L, the knee portion Q2L, the waist portion Q3R, Q3L and the scapula portion Q4R, Q4L is given substantially at the same time.
- stimulation to the shin portion P1R, P1L, the knee portion, P2L, the waist portion P3R, P3L and the chest portion P4R, P4L is given substantially at the same time.
- the knee portion Q2R is stimulated, and after the predetermined time elapses, the knee portion P2R is stimulated.
- the user By stimulating each part of the user at the timing as shown in FIG. 15, the user can perceive the state of gradually sinking from the back to the bottom of the water while lying down, which can increase the sense of presence. can.
- the last stimulus to the right knee portion Q2R and P2R is generated because, as shown in FIG. 6, the person is lying down with the right knee upright.
- the user as shown in FIG. 7, that is, the user sitting sideways may be given a tactile stimulus at the timing shown in FIG. This is because the height position of each part of the user is the same in the state shown in FIG. 7 and the state shown in FIG. 4, and it is preferable to give the tactile stimuli transmitted in order from the bottom to the top in the same order. Is.
- the presentation control device 31 is a tactile signal worn by the user.
- the presentation of the tactile stimulus by the reproduction device 32 may be stopped.
- the tactile presentation may be canceled when the reference posture and the user's posture are significantly different from each other. Specifically, when the discrepancy between the IMU data (reference data) when the user is in the reference state and the acquired IMU data exceeds a predetermined threshold value, it is determined that the tactile signal cannot be adjusted by the adjustment process. Stop tactile presentation.
- the creator of the tactile signal may specify one or more reference postures (for example, the posture shown in FIG. 4, the posture shown in FIG. 5, the posture shown in FIG. 6, the posture shown in FIG. 7, and the like). It may be determined that the tactile signal cannot be adjusted when the posture is different from the reference posture.
- the presentation of the tactile stimulus may be stopped by canceling the transmission of the tactile signal from the presentation control device 31 to the tactile signal reproducing device 32.
- the presentation of the tactile stimulus may be stopped by the tactile signal reproducing device 32 receiving the tactile signal transmitted from the presentation control device 31 not reproducing the tactile signal.
- the presentation of the tactile stimulus when the presentation of the tactile stimulus is stopped, the presentation of all the tactile stimuli to the user may be stopped, but the user may be able to enjoy the content at the minimum by stopping a part of the tactile stimulus.
- the user wears a vest type tactile signal reproduction device 32C, a tactile signal reproduction device 32D to be attached to the right arm, and a tactile signal reproduction device 32E to be attached to the left arm.
- the presentation control device 31 provides information indicating that the user has started operating the smartphone, information indicating that the user's line of sight has deviated from the screen 24, information indicating that the screen of the smartphone has become bright, and the like. Acquire as status information. These information may be acquired from the tactile signal reproduction device 32 worn by the user, may be acquired from the smartphone operated by the user, or may be acquired from an image pickup device installed near the screen 24 or the like. You may get it from such as.
- the presentation control device 31 stops the tactile presentation by the tactile signal reproducing device 32D attached to the right arm portion used for operating the small terminal in response to the acquisition of these state information. On the other hand, the tactile presentation by the tactile signal reproducing device 32C and the tactile signal reproducing device 32E continues.
- the user can continue the operation of the smartphone without receiving the tactile stimulus that interferes with the operation of the smartphone, and can enjoy the tactile stimulus according to the content.
- FIG. 17 shows an example in which a user wears a vest type tactile signal reproducing device 32A provided with one vibrating portion 52C on the waist portion on the back side.
- FIG. 18 shows an example in which a user wears a vest type tactile signal reproducing device 32B in which one vibrating portion 52 is provided on each of the left and right waist portions.
- the vibrating portion 52 of the left hip will be referred to as a vibrating portion 52L
- the vibrating portion 52 of the right hip will be referred to as a vibrating portion 52R.
- the user wearing any of the tactile signal reproduction devices 32 is in a state of lying down with the left half of the body down.
- the vibrating section 52L on the left hip is in close contact with the user's body and the sofa (ground).
- the vibration unit 52L is vibrated with the same vibration intensity as the vibration unit 52C and the vibration unit 52R, the user feels a vibration stronger than the vibration of the vibration unit 52R on the right hip from the vibration unit 52L.
- the two vibrating units 52R and 52L provide the tactile stimulus to the user, whereas in the embodiment shown in FIG. 17, the tactile stimulus is provided to the user by one vibrating unit 52C. be. Therefore, it is conceivable to vibrate the vibrating portion 52C with a vibration intensity stronger than that of the vibrating portion 52R. Specifically, it is conceivable to increase the vibration intensity in the order of the vibrating portion 52L ⁇ vibrating portion 52R ⁇ vibrating portion 52C.
- such an adjustment process is a process required according to the combination of the user's posture and device information (for example, the position of the vibrating unit 52). That is, if the combination of the user's posture and the device information is different, different adjustment processes are required.
- the vibration timing may be adjusted according to the position of the vibrating portion 52. Specifically, the timing at which the vibrating unit 52C is vibrated with respect to the user A wearing the tactile signal reproducing device 32A shown in FIG. 17 and the user B wearing the tactile signal reproducing device 32B shown in FIG. 18
- the timing of vibrating the vibrating portions 52L and 52R is set to the same timing, the timing at which the user A feels the tactile stimulus on the left and right hip portions may be delayed from that of the user B. This is because it takes time for the tactile stimulus from the vibrating portion 52C to reach the left and right hip portions of the user.
- the tactile signal reproducing device 32 in which the vibrating portion 52 is arranged on the back side and the tactile signal in which the vibrating portion 52 is not arranged on the back side but arranged on the abdominal side.
- the reproduction device 32 it is conceivable to shift the generation timing of the vibration so as to vibrate the vibrating portion 52 on the abdominal side later in consideration of how the vibration is transmitted.
- the characteristics of the vibration unit 52 may be considered. For example, consider the case where the reproducible vibration intensity differs depending on the type of the vibrating unit 52. If a large vibration is generated even though the vibration unit 52 has a low reproducible vibration strength, it becomes impossible to generate the large vibration when a scene that should generate a larger vibration arrives. There is a risk that it will end up. As a result, the richness of expression due to the strength of vibration is lost, and it becomes impossible for the user to experience an appropriate tactile stimulus.
- the characteristics of the vibrating unit 52 may be taken into consideration by considering the frequency characteristics of the vibrating unit 52. For example, when the attenuation of the high frequency band in the vibration output from a certain vibration unit 52 is large, it may be dealt with by increasing the gain of the high frequency band.
- the strength of the tactile signal transmitted to the user B far from the screen 24 is weaker than the strength of the tactile signal transmitted to the user A who is close to the screen 24.
- the user can experience an appropriate tactile stimulus according to the distance to the screen 24.
- the user A and the user B are enjoying the content in the reference posture (for example, the posture of sitting on a chair), while the user C is in the standing posture and moving away from the screen 24.
- the reference posture for example, the posture of sitting on a chair
- the user C is in the standing posture and moving away from the screen 24.
- the strength of the tactile signal is further weakened or the presentation of the tactile stimulus is stopped for the user C as compared with the user B.
- the communication control unit 46 of the presentation control device 31 can perform transmission control for performing various notifications to the user.
- the communication control unit 46 performs a process of notifying that the tactile stimulus provided to the user is not based on the reference tactile signal but based on the adjusted tactile signal after adjustment. This process is executed in response to the adjustment process performed by the adjustment process unit 45.
- the user can recognize that he / she is experiencing the tactile stimulus that has been optimally adjusted, and can also recognize that the posture is different from the reference posture. ..
- the communication control unit 46 performs a process of notifying the user that the tactile presentation has been stopped. This process is executed, for example, when it is detected that the user is not visually recognizing the screen 24, or when the user's posture deviates too much from the reference posture, as described in the second example of the adjustment process. Will be done.
- the user can recognize that the presentation of the tactile stimulus has been stopped, and can also recognize that his / her posture is too deviated from the reference posture.
- Such various notification processes are realized by transmitting the packetized data from the presentation control device 31 to the tactile signal reproduction device 32.
- FIG. 20 shows an example of the packet data structure.
- Packet data is configured with a header, payload and footer.
- the header may include various control information, synchronization signals, and the like.
- the footer may include various control information, parity information, and the like.
- the tactile signal data is stored in the payload.
- data indicating the content of the notification can be stored in the payload and transmitted.
- the notification content may be included in the packet data generated for transmitting the tactile signal. Further, in that case, if the data indicating the notification content is stored in the payload, the packet data becomes redundant, so that the packet data may be stored in the header or footer.
- the reproduction time of the tactile stimulus reproduced by one packet data is very short, such as several msec to several tens of msec. Therefore, in order to present a tactile stimulus over a long time such as several seconds, a plurality of packets such as several to several tens are used.
- Such a method can be applied not only to the transmission data in the notification processing but also to various control data transmitted from the presentation control device 31 to the tactile signal reproduction device 32.
- the tactile signal reproduction device 32 that has received the control information and the notification information by such a method can restore the received data by executing a process of restoring the control information and the notification information from a plurality of packet data. ..
- the above-mentioned presentation control device 31 and the tactile signal reproduction device 32 have a configuration as a computer device in order to have each of the above-mentioned functions. It should be noted that each device or device does not have to have all the configurations shown below, and may have only a part thereof.
- the CPU 71 of each device loads a program stored in a non-volatile memory unit 74 such as a ROM 72 or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a storage unit 79 into a RAM 73.
- ROM 72 or an EEPROM Electrical Erasable Programmable Read-Only Memory
- the RAM 73 also appropriately stores data and the like necessary for the CPU 71 to execute various processes.
- the CPU 71, ROM 72, RAM 73, and non-volatile memory unit 74 are connected to each other via a bus 83.
- An input / output interface 75 is also connected to the bus 83.
- An input unit 76 including an operator and an operation device is connected to the input / output interface 75.
- various controls and operation devices such as a keyboard, a mouse, a key, a dial, a touch panel, a touch pad, and a remote controller are assumed.
- the user's operation is detected by the input unit 76, and the signal corresponding to the input operation is interpreted by the CPU 71.
- a display unit 77 made of an LCD or an organic EL panel and an audio output unit 78 made of a speaker or the like are connected to the input / output interface 75 as one or as a separate body.
- the display unit 77 is a display unit that performs various displays, and is composed of, for example, a display device provided in a housing of a computer device, a separate display device connected to the computer device, and the like.
- the display unit 77 executes the display of various images for image processing, moving images to be processed, and the like on the display screen based on the instruction of the CPU 71. Further, the display unit 77 displays various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface) based on the instruction of the CPU 71.
- GUI Graphic User Interface
- the input / output interface 75 may be connected to a storage unit 79 composed of a hard disk, a solid-state memory, or the like, or a communication unit 80 composed of a modem or the like.
- the communication unit 80 performs communication processing via a transmission line such as the Internet, wired / wireless communication with various devices, bus communication, and the like.
- a drive 81 is also connected to the input / output interface 75, if necessary, and a removable storage medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted.
- the drive 81 can read data files such as image files and various computer programs from the removable storage medium 82.
- the read data file is stored in the storage unit 79, and the image and sound included in the data file are output by the display unit 77 and the sound output unit 78. Further, the computer program or the like read from the removable storage medium 82 is installed in the storage unit 79 as needed.
- software for the processing of the present embodiment can be installed via network communication by the communication unit 80 or removable storage medium 82.
- the software may be stored in the ROM 72, the storage unit 79, or the like in advance.
- the information processing device constituting the presentation control device 31 and the tactile signal reproduction device 32 is not limited to a single computer device as shown in FIG. 21, but is configured by systematizing a plurality of computer devices. You may.
- the plurality of computer devices may be systematized by a LAN or the like, or may be arranged in a remote place by a VPN or the like using the Internet or the like.
- the plurality of information processing devices may include an information processing device as a server group (cloud) that can be used by a cloud computing service.
- the CPU 71 of the presentation control device 31 executes the information acquisition process in step S101 of FIG.
- information acquisition process information necessary for executing the various processes described above is acquired.
- An example is shown in FIG.
- the CPU 71 of the presentation control device 31 acquires device information in step S201.
- the device information is a process for acquiring information such as the number of vibration units 52, the arrangement position, and the frequency characteristics of the tactile signal reproduction device 32.
- the state information includes not only the posture information of the user but also the information indicating the orientation of the body and the information indicating whether or not the user is leaning against the backrest.
- the CPU 71 of the presentation control device 31 acquires the visual recognition information in step S203.
- the visual recognition information is information indicating whether or not the user is visually recognizing the screen 24.
- the distance information is, for example, information indicating the distance between the user and the screen 24.
- the CPU 71 of the presentation control device 31 that has completed the information acquisition process determines in step S102 of FIG. 22 whether or not the tactile signal reproduction device 32 used by the user is the same as the reference device.
- the CPU 71 of the presentation control device 31 proceeds to step S104 without performing the process of step S103.
- the CPU 71 of the presentation control device 31 executes the adjustment process according to the device information in step S103. For example, for a user wearing a tactile signal reproducing device 32 having only one vibrating unit 52 for a reference device provided with two vibrating units 52, two reference tactile signals are provided. After synthesizing the two signals, an adjusted tactile signal with increased signal strength is generated.
- step S104 the CPU 71 of the presentation control device 31 determines whether or not the adjustment process based on the state information is permitted. For example, if the creator of the reference tactile signal does not permit the adjustment process, or if the user wearing the tactile signal reproduction device 32 does not permit the adjustment process, the CPU 71 of the presentation control device 31 steps. The process proceeds to step S108 in FIG. 23 without executing the processes of S105, step S106, and step S107.
- step S108 If the user's state is the same as the reference state, the process proceeds to step S108.
- the CPU 71 of the presentation control device 31 performs the adjustment process according to the state information in step S106, and notifies the user that the adjustment has been performed in step S107. Perform notification processing.
- step S106 and step S107 are not executed. You may proceed to step S108.
- the CPU 71 of the presentation control device 31 determines in step S108 of FIG. 23 whether or not adjustment processing according to the distance information is necessary. For example, when the reference tactile signal is created assuming a tactile stimulus given to a user at a position 10 m away from the screen 24, and the user wearing the tactile signal reproduction device 32 is at a position 2 m away from the screen 24. Is determined to require adjustment according to the distance information.
- the CPU 71 of the presentation control device 31 executes the adjustment process according to the distance information in step S109.
- the timing of presenting the tactile stimulus may be advanced in consideration of the propagation time of the impact or the like, so that the discomfort of the tactile stimulus may be reduced as much as possible.
- step S110 the CPU 71 of the presentation control device 31 proceeds to step S110 without performing the process of step S109.
- the CPU 71 of the presentation control device 31 transmits a tactile presentation signal in step S110.
- the reference tactile signal or the adjusted tactile signal is transmitted from the presentation control device 31 to the tactile signal reproducing device 32.
- the CPU 71 of the presentation control device 31 determines whether or not the adjustment process is permitted in step S104, it determines whether or not the reference state and the user state are the same in step S105. Then, when it is determined that the user's state is different from the reference state, the CPU 71 of the presentation control device 31 determines in step S111 whether or not the reference tactile signal can be adjusted.
- the CPU 71 of the presentation control device 31 proceeds to the process of step S108 without executing the processes of steps S106 and S107.
- the tactile signal may not be transmitted from the presentation control device 31 to the tactile signal reproduction device 32, or the control information indicating that the tactile signal transmitted from the presentation control device 31 is not reproduced. May be added so that the tactile signal is not reproduced by the tactile signal reproducing device 32.
- the CPU 71 of the presentation control device 31 executes the adjustment process according to the state information in step S106, and performs the notification process in step S107.
- 26 and 27 show the configurations of the presentation control device 31 and the tactile signal reproduction device 32 in the modified example.
- the presentation control device 31 includes an information acquisition unit 41, a tactile signal acquisition unit 42, and a communication control unit 46. Further, as shown in FIG. 27, the tactile signal reproduction device 32 includes a determination processing unit 43, an identification information setting unit 44, an adjustment processing unit 45, a reproduction control unit 51, a vibration unit 52, a posture detection unit 53, and a communication control unit 54. I have.
- the presentation control device 31 transmits the reference tactile signal to the tactile signal reproducing device 32.
- the tactile signal reproduction device 32 acquires the posture information of the user detected by the posture detection unit 53, the information detected by the pressure sensitive element included in the vibration unit 52, and the like. Further, the tactile signal reproduction device 32 determines whether or not to perform the adjustment process based on the various acquired information.
- the adjustment processing unit 45 performs the adjustment processing on the reference tactile signal acquired from the presentation control device 31 to generate the adjustment tactile signal. Then, the reproduction control unit 51 realizes the presentation of the tactile stimulus by outputting the reference tactile signal and the adjusted tactile signal to the vibrating unit 52.
- the presentation control device 31 increases the processing load when there are a large number of tactile signal reproduction devices 32 for which the tactile signal is transmitted. According to this modification, since the execution feasibility determination and the adjustment process of the tactile signal adjustment process are performed by each tactile signal reproduction device 32, it is possible to prevent the processing load from being concentrated on the presentation control device 31. can.
- the above-mentioned techniques can also be incorporated into game systems and the like.
- a game system including a game machine, a display device, and a game controller
- the presentation control device 31 described above is replaced with a game machine
- the tactile signal reproduction device 32 described above is replaced with a game controller to provide a more realistic feeling. It is possible to enjoy the increased game content.
- a tactile stimulus that matches the user's operation style by performing a process of determining whether or not the game controller is strongly gripped. Specifically, the tactile stimulus presented to the user who plays the game by firmly grasping the game controller is weakened. In addition, the tactile stimulus presented to the user who plays the game by lightly grasping the game controller is strengthened. As a result, it is possible to provide the user with a tactile stimulus of the same intensity regardless of the play style.
- the state information including the posture information of the user may be acquired from the tactile signal reproduction device 32, or may be acquired from an information processing terminal (for example, a smartphone) possessed by the user (see FIG. 28).
- the user's state information may be acquired by storing the posture of the smartphone when the user takes the reference posture and receiving the relative change amount from the smartphone from the smartphone.
- an image pickup device that captures the user is placed, image analysis processing is performed on the captured image acquired from the image pickup device, and the posture of the user is estimated based on the obtained information.
- Information may be obtained.
- the state is estimated by estimating the posture information of the user according to the positional relationship between the tactile signal reproducing devices 32 and the positional relationship between the vibrating units 52. Information may be obtained. For example, the distance between the tactile signal reproducing device 32 attached to the knee portion of the user and the tactile signal reproducing device 32 attached to the chest differs between the standing state and the sitting state. By analyzing this difference, it may be determined whether the user is standing or sitting.
- the vibration unit 52 is provided as the tactile presentation unit in the tactile signal reproduction device 32
- other examples can be considered.
- a blower for providing a stimulus by a blast to a user a device for providing a stimulus by water, water vapor, or the like to a user, and an electrode device for providing a stimulus by an electric stimulus to the user.
- these various devices and devices included in the tactile signal reproducing device 32 can provide different tactile stimuli according to the user's state information and the like.
- a blower it is possible to present an appropriate tactile stimulus according to the posture of the user by changing the operating blower and adjusting the amount of blown air and the timing of blowing air.
- a device that provides water or the like for example, a device provided in the chair-type tactile signal reproduction device 32
- the discharge direction, amount, release timing, etc. of water (water vapor) can be determined according to the posture of the user.
- an appropriate tactile stimulus can be provided to the user.
- the electrode device it is possible to present an appropriate tactile stimulus according to the posture of the user by adjusting the place where the electric stimulus is applied, the amount of energization, the generation pattern of the electric stimulus, and the like.
- the presentation control device 31 as the signal processing device of the present technology is for performing tactile presentation to the user in the reference state with the information acquisition unit 41 that acquires the state information indicating the user's state including the posture of the user.
- the tactile signal acquisition unit 42 that acquires the reference tactile signal of the above, and the adjustment processing unit 45 that generates the adjustment tactile signal by adjusting the reference tactile signal based on the state information are provided.
- Different postures of the user also result in different tactile stimuli experienced by the user.
- the adjusted tactile signal according to the user's state is generated. This makes it possible to present the user with a tactile sensation based on an appropriate tactile signal.
- the presentation control device 31 as such a signal processing device is suitable for, for example, a tactile signal reproduction device 32 worn on a user's arm or the like.
- the position and orientation of the user's arm are likely to differ from user to user, and normally there are very few situations in which an appropriate tactile presentation can be performed according to the orientation and position of the user's arm. ..
- a tactile signal reproducing device 32 it is possible to appropriately provide the intended tactile presentation to the user's arm regardless of the posture of the user's arm. .. This makes it possible to enhance the sense of presence and immersiveness.
- the determination processing unit 43 for determining whether or not the adjustment can be executed may be provided.
- the tactile stimulus appropriate for the user may differ depending on the posture of the user or the like, or may be the same regardless of the posture of the user. According to this configuration, it is possible to appropriately switch between the case where the adjustment is performed based on the user's state information and the case where the adjustment is not performed. Therefore, it is possible to make adjustments only for the tactile signal to be adjusted according to the posture.
- the determination processing unit 43 may determine the determination result of whether or not the execution is possible based on the operation of the user. Whether or not to adjust the tactile signal may be determined not only automatically according to the type of the tactile signal but also by the user. According to this configuration, since the user himself / herself can decide whether or not to adjust the tactile signal, it is possible to adapt to the user's preference and the like.
- the determination processing unit 43 may determine whether or not adjustment is necessary due to the difference between the posture of the user and the reference state. This makes it possible to appropriately determine whether or not the reference tactile signal should be adjusted, taking into account the posture of the user. Therefore, an appropriate tactile stimulus can be given according to the posture of the user. Further, even if the posture of the user and the reference state are slightly different, it may be determined that the adjustment is unnecessary. In such a case, the execution of the adjustment process is avoided, so that the processing load of the presentation control device 31 is increased. Is reduced.
- the communication control unit 46 for transmitting the tactile signal to the tactile signal reproducing device 32 (32A, 32B, 32C, 32D, 32E) for reproducing the tactile signal.
- the unit 46 may perform transmission control so that the tactile signal reproduction device 32 does not reproduce the tactile signal when it is determined that the adjustment is impossible.
- the tactile signal reproduction device 32 does not reproduce the tactile signal when it is determined that the adjustment is impossible.
- the communication control unit 46 may be set not to transmit the tactile signal when it is determined that the adjustment is impossible. As a result, the processing load of the tactile signal transmission processing is reduced. Therefore, when a tactile signal is transmitted to each tactile signal reproduction device 32 (32A, 32B, 32C, 32D, 32E) used by a plurality of users, the processing task overflows and the transmission processing is appropriate timing. It is possible to avoid that it will not be executed by. Further, the processing load related to the reception processing and the reproduction processing of the tactile signal reproduction device 32 on the receiving side is also reduced.
- the communication control unit 46 may transmit the tactile signal by adding control information so that the tactile signal is not reproduced when it is determined that the adjustment is impossible. good.
- the presentation control device 31 as a signal processing device that adjusts the tactile signal can uniformly transmit the tactile signal regardless of the necessity of adjustment. For example, when transmitting a tactile signal to a plurality of tactile signal reproducing devices, it is not necessary to control so that the tactile signal is not transmitted only to a specific tactile signal worn by a user who takes a posture significantly different from the reference state. Then, whether or not to execute the reproduction of the tactile signal is performed by the tactile signal reproduction device 32 that has received the tactile signal. Therefore, the processing load of the signal processing device can be reduced.
- the communication control unit 46 that controls transmission to transmit the tactile signal to the tactile signal reproduction device 32 (32A, 32B, 32C, 32D, 32E) that reproduces the tactile signal.
- the information acquisition unit 41 acquires visual information indicating whether or not the user is visually recognizing the image to be visually recognized by the user, and the communication control unit 46 visually recognizes the information indicating that the user is not visually recognizing the image.
- transmission control may be performed so that the tactile signal reproduction is not performed by the tactile signal reproduction device 32.
- the user may experience the tactile stimulus in various situations. And, in some cases, it may or may not be concentrated on the content.
- the presentation of the tactile stimulus to such a user can be interrupted or stopped so as not to interfere with another action that the user is performing with the viewing of the content.
- a tactile stimulus is provided to such a user, there is a risk that the user's behavior may be hindered, such as not being able to write accurately.
- the communication control unit 46 for transmitting the tactile signal to the tactile signal reproducing device 32 (32A, 32B, 32C, 32D, 32E) for reproducing the tactile signal is provided.
- the unit 46 may perform transmission control for notifying that the adjustment has been made.
- the user can know that the reference tactile signal is adjusted. Therefore, it is possible to understand that it is not necessary to re-recognize the meaning of the tactile stimulus in consideration of one's posture and posture. That is, by experiencing the tactile stimulus obediently, it is possible to realize that the tactile stimulus intended by the creator can be received, so that a sense of presence and immersiveness can be experienced.
- the adjustment process unit 45 may make an adjustment to change the intensity of the tactile presentation in the adjustment of the reference tactile signal. This makes it possible to present a tactile sensation of appropriate strength according to the state of the user.
- the adjustment process unit 45 may perform adjustment according to the tactile signal reproduction device 32 (32A, 32B, 32C, 32D, 32E) worn by the user. ..
- the adjustment process unit 45 is a tactile presentation unit (vibration unit 52, 52C, 52R, 52L) included in the tactile signal reproduction device 32 (32A, 32B, 32C, 32D, 32E). ) May be adjusted in intensity and timing of tactile presentation according to the position and state information. As a result, even if each user wears a tactile signal reproducing device 32 in which the vibrating portion 52 is arranged at a different position, an appropriate tactile signal is generated for each user. Therefore, the user can enjoy the tactile stimulus without being aware of not only the difference in posture but also the difference in the arrangement position of the vibrating portion 52. Further, depending on the position of the vibration unit 52, the strength when the user experiences the vibration may differ.
- the information acquisition unit 41 acquires the distance information between the display device (screen 24) on which the image to be visually recognized by the user is displayed and the user, and the adjustment process unit 45 acquires the distance information. Adjustments may be made to change the intensity of the tactile presentation according to the distance information provided. When the user enjoys the tactile stimulus while watching the image, if the appearance of the image is different, the optimum tactile stimulus is likely to be different. According to this configuration, it is possible to increase the strength of the tactile presentation as the user is closer to the display device, so that the sense of presence and immersiveness felt by the user can be further enhanced.
- the information acquisition unit 41 may acquire state information each time the posture of the user changes. As the user's posture changes, so does the optimal tactile stimulus. By acquiring the latest state information of the user in response to a change in posture, it is possible to continue to provide the optimum tactile stimulus even in a situation where the user is moving intermittently.
- the present technology can also adopt the following configurations.
- An information acquisition unit that acquires status information indicating the user's status, including the user's posture,
- a tactile signal acquisition unit that acquires a reference tactile signal for presenting a tactile sensation to a user in a reference state, and a tactile signal acquisition unit.
- a signal processing device including an adjustment processing unit that generates an adjustment tactile signal by adjusting the reference tactile signal based on the state information.
- the signal processing apparatus according to (1) above comprising a determination processing unit for determining whether or not the adjustment can be executed.
- the signal processing device adds control information and transmits a tactile signal so that the tactile signal is not reproduced when it is determined that the adjustment is impossible. .. (8) It is equipped with a communication control unit that controls transmission to transmit tactile signals to a tactile signal reproduction device that reproduces tactile signals.
- the information acquisition unit acquires visual information indicating whether or not the user is visually recognizing the image to be visually recognized by the user.
- the communication control unit performs transmission control so that the tactile signal is not reproduced by the tactile signal reproduction device when the information indicating that the user is not visually recognizing the image is acquired as the visual information.
- the signal processing device according to any one of the above (1) to (7).
- (9) It is equipped with a communication control unit that controls transmission to transmit tactile signals to a tactile signal reproduction device that reproduces tactile signals.
- the signal processing device according to any one of (1) to (8) above, wherein the communication control unit performs transmission control for notifying that adjustment has been made when transmitting the adjustment tactile signal.
- the adjustment processing unit makes adjustments for changing the intensity of the tactile presentation in the adjustment of the reference tactile signal.
- the adjustment processing unit performs the adjustment according to the tactile signal reproduction device worn by the user.
- the signal processing device (12) The signal processing device according to (11) above, wherein the adjustment processing unit adjusts the intensity and timing of the tactile presentation according to the position of the tactile presentation unit included in the tactile signal reproduction device and the state information.
- the information acquisition unit acquires distance information between the display device on which the image to be visually recognized by the user is displayed and the user.
- the signal processing device according to any one of (10) to (12) above, wherein the adjustment processing unit makes adjustments to change the intensity of the tactile presentation according to the acquired distance information.
- the information acquisition unit acquires the state information each time the posture of the user changes.
- Processing to acquire state information indicating the user's state including the user's posture The process of acquiring the reference tactile signal for presenting the tactile sensation to the user in the reference state, and A signal processing method in which a signal processing device executes a process of generating an adjusted tactile signal by adjusting the reference tactile signal based on the state information.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Psychiatry (AREA)
- Computer Networks & Wireless Communication (AREA)
- Databases & Information Systems (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
同一のコンテンツを視聴している各ユーザに対しては、同様の触覚刺激を提供することが望ましい。
しかし、各種の要因によってユーザごとに触覚刺激の感じ方は異なる。
このような問題点に対して、下記特許文献1においては、触覚提示デバイスが所望のとおりに動作しているかを確認した上で必要であれば触覚刺激をユーザに提供するための触覚信号を調整する技術が開示されている。
しかし、特許文献1に記載の手法では、ユーザ(被体験者)が自身で触覚刺激を体感して確認した上で調整する形となっており、必ずしも制作者の意図を反映した調整がなされるとは限らない。
ユーザの姿勢が異なれば、ユーザが体感する触覚刺激も異なる。本構成によれば、姿勢を含めたユーザの状態を示す状態情報に基づいて基準触覚信号を調整することにより、ユーザの状態に合わせた調整触覚信号が生成される。
ユーザの姿勢等に応じて適切な触覚刺激が異なる場合とそうでない場合がある。本構成によれば、ユーザの状態情報に基づく調整を行う場合と行わない場合とを適宜切り換えることが可能となる。
触覚信号の調整を行うか否かについては、触覚信号の種別等に応じて自動で決定するだけでなくユーザが決定することも考えられる。
これにより、調整処理部は、基準触覚信号ごとに識別情報を確認することにより調整を行うか否かを決定することができる。
これにより、基準触覚信号を調整すべきか否かについてユーザの姿勢を加味して適切に判定することができる。
例えば、ユーザの状態(姿勢など)が基準状態と比較してあまりにもかけ離れている場合には、基準触覚信号の調整をしても適切な触覚刺激をユーザに体感させることができないことがある。
調整が不能とされた場合に触覚信号の再生が行われないようにすることで、ユーザに触覚刺激を与えないようにすることができる。
これにより、触覚信号の送信処理の処理負担が軽減される。
これにより、触覚信号の調整を行う信号処理装置は、調整の要否によらず一律触覚信号の送信処理を行うことができる。例えば、複数の触覚信号再生機器に対して触覚信号を送信する場合に、基準状態と大きく異なる姿勢をとるユーザが装着する特定の触覚信号にだけ触覚信号を送信しないように制御する必要がない。
ユーザは、各種の状況で触覚刺激を体感することが考えられる。そして、その中には、コンテンツに集中している場合と集中していない場合がある。例えば、映像と共に触覚刺激を楽しむユーザが、コンテンツに集中しているか否かを判定する手法としては、映像をきちんと視聴しているか否かを判定することが考えられる。
これにより、ユーザは、基準触覚信号が調整されていることを把握することができる。
これにより、ユーザの状態に応じて適切な強度の触覚提示を行うことが可能となる。
これにより、ユーザの状態の違いだけでなく装着している触覚信号再生機器の違いに応じて適切な調整がなされた触覚信号を送信することが可能となる。
これにより、振動部の配置位置が異なる触覚信号再生機器を各ユーザが着用していたとしても、ユーザごとに適切な触覚信号が生成される。また、前記触覚信号再生機器が備える触覚提示部の再生周波数特性に応じて前記触覚提示信号の周波数特性を調整してもよい。
ユーザが映像を見ながら触覚刺激を楽しむ場合において、映像の見え方が異なれば、最適な触覚刺激も異なる可能性が高い。
ユーザの姿勢が変化すれば、最適な触覚刺激も異なる。
<1.システム構成>
<2.調整処理>
<2-1.調整処理の第1例>
<2-2.調整処理の第2例>
<2-3.調整処理の第3例>
<2-4.調整処理の第4例>
<3.通知処理>
<4.コンピュータ装置の構成>
<5.処理例>
<6.変形例>
<7.まとめ>
<8.本技術>
<1.システム構成>
本実施の形態では、制作者によって作成された触覚信号に基づいて触覚刺激をユーザに提供する信号処理システムについて説明する。
なお、以下の説明においては、映画コンテンツの臨場感をより向上させるためにユーザに触覚刺激を提供する例を挙げるが、本技術の実施はこれに限られることはない。例えば、音楽や声などの音響データと触覚刺激を組み合わせることによりユーザに新たな体験を提供するものであってもよい。また、ゲームコンテンツが備える映像データと音響データに合わせて触覚刺激を提供することによりゲームをより楽しめるものにするものであってもよい。もちろん、映像データや音響データ以外の情報と触覚刺激を組み合わせてユーザに提供するものであってもよい。
エンタテインメントシステム1は、映画コンテンツにおける映像及び音声をユーザに視聴させるシネマシステム2と、触覚信号に基づく触覚刺激をユーザに提供するための信号処理を行う信号処理システムとしての提供システム3とを備えている。
触覚信号再生機器32は、再生制御部51と、振動部52と、装着しているユーザの姿勢を検出する姿勢検出部53と、無線通信を介して触覚信号を受信する通信制御部54と、を備えている(図3参照)。
ユーザの状態情報を定期的に取得する場合には、状態情報を送信する機器において送信を行うか否かを判定する処理などを実行しなくて済むため、処理負担の軽減が図られる。
また、ユーザの状態が変化するごとにユーザの状態情報を取得する場合には、必要な情報を必要なタイミングで取得することとなるため、通信量の削減や通信に係る処理負担の軽減を図ることができる。
更に、ユーザの姿勢変化によらず一度だけ状態情報を取得する場合には、状態情報の送信や受信に係る通信量や処理負担を大幅に軽減させることができる。
具体的には、触覚信号再生機器32からIMUデータを取得し、情報取得部41は、受信したIMUデータに基づいてユーザの姿勢などを推定することにより状態情報を取得してもよい。或いは、触覚信号再生機器32においてIMUデータからユーザの姿勢を推定する処理が実行されることにより、触覚信号再生機器32から状態情報を取得してもよい。
なお、デバイス情報としてデバイスの種類ごとの特性情報をDBに記憶しておき、触覚信号再生機器32から取得した型番情報などに基づいて情報取得部41がDBからデバイス情報を取得してもよい。
デバイス情報は、想定していた特定の触覚信号再生機器32(基準デバイス)とユーザが使用中の触覚信号再生機器32との違いを特定するための情報ともいえる。そして、ユーザが使用中の触覚信号再生機器32が基準デバイスと異なっている場合には、後述する調整処理を行う。
視聴情報を取得するための画像は、例えば、スクリーン24の近傍に配置された撮像装置で撮像された画像など、各種のものを用いることができる。
基準触覚信号ごととは、映画における一場面においてユーザに提供する一連の基準触覚信号ごとであってもよいし、一つのパケットデータに含まれる基準触覚信号ごとであってもよい。なお、一つのパケットデータには、例えば、再生時間長が数msec~数百msecなど比較的短い時間とされた触覚信号が含まれている。
提示制御装置31の調整処理部45が実行する触覚信号の調整処理についていくつかの例をあげて説明する。
調整処理の第1例では、映画の一場面に登場する人物の足下から上方に向かって徐々に水面が上がってくる状況においてユーザに提供する触覚刺激について説明する。
具体的に添付図を参照して説明する。
なお、触覚信号再生機器32が備える振動部52の位置や個数や特性が想定と異なる場合には、基準触覚信号の調整が必要となる場合がある。このような調整処理を提示制御装置31で行う場合には、基準触覚信号を調整した調整触覚信号が触覚信号再生機器32に送信される。また、調整処理を触覚信号再生機器32側で行う場合には、基準触覚信号が触覚信号再生機器32に送信される。
一例として、ユーザが触覚信号再生機器32を着用或いは使用することにより、ユーザの前面側(腹側)に8箇所の刺激を与えることが可能とされている。即ち、ユーザの左右のすね部分P1R,P1L、左右のひざ部分P2R,P2L、左右の腰部分P3R,P3L、左右の胸部分P4R,P4Lにそれぞれ触覚刺激を与えることが可能とされている(図11参照)。
その後、ユーザの胸部分P4R,P4L及び肩胛骨部分Q4R,Q4Lへの刺激を略同時に与える。
この場合には、図14に示すように、すね部分P1R,P1L,Q1R,Q1Lへの刺激を与えた後、所定時間後にひざ部分P2R,P2L,Q2R,Q2Lへの刺激を与える。更に所定時間後に腰部分P3R,P3L,Q3R,Q3Lへの刺激を与え、所定時間経過後にユーザの胸部分P4R,P4L及び肩胛骨部分Q4R,Q4Lへの刺激を略同時に与える。
図14に示すようなタイミングでユーザの各部に刺激を与えることにより、即ち、ひざ部分P2R,P2L,Q2R,Q2Lと腰部分P3R,P3L,Q3R,Q3Lへの刺激を時間差で与えることにより、ユーザは、立っている状態において足下から水面が上がってくる状態を知覚することができ、臨場感を増すことができる。
この場合には、図15に示すように、すね部分Q1R,Q1L、ひざ部分Q2L、腰部分Q3R,Q3L及び肩胛骨部分Q4R,Q4Lへの刺激を略同時に与える。
次に、所定時間経過後にすね部分P1R,P1L、ひざ部分,P2L、腰部分P3R,P3L及び胸部分P4R,P4Lへの刺激を略同時に与える。
更に所定時間経過後に、ひざ部分Q2Rへの刺激を与え、所定時間経過後にひざ部分P2Rへの刺激を与える。
図15に示すようなタイミングでユーザの各部に刺激を与えることにより、ユーザは、寝転がっている状態において背中から徐々に水面下に沈んでいく状態を知覚することができ、臨場感を増すことができる。
調整処理の第2例では、ユーザの状態情報に応じて触覚刺激の提示を停止させる例などを説明する。
例えば、提示制御装置31から触覚信号再生機器32への触覚信号の送信を取りやめることにより触覚刺激の提示が停止されてもよい。
或いは、提示制御装置31から送信された触覚信号を受信した触覚信号再生機器32が当該触覚信号の再生を行わないことにより触覚刺激の提示が停止されてもよい。
調整処理の第3例では、ユーザの状態情報とデバイス情報の組み合わせに応じて基準触覚信号を調整する場合について説明する。
具体的に添付図を参照して説明する。
また、図18は、左右両側の腰部分にそれぞれ一つの振動部52が設けられたベスト型の触覚信号再生機器32Bをユーザが装着している例を示したものである。左腰の振動部52を振動部52Lとし、右腰の振動部52を振動部52Rとして説明する。
従って、振動部52Rよりも強い振動強度で振動部52Cを振動させることが考えられる。具体的には、振動部52L<振動部52R<振動部52Cの順で振動強度を強くすることが考えられる。
具体的には、図17に示す触覚信号再生機器32Aを装着しているユーザAに対して振動部52Cを振動させるタイミングと、図18に示す触覚信号再生機器32Bを装着しているユーザBに対して振動部52L,52Rを振動させるタイミングを同じタイミングとしてしまうと、ユーザAが左右の腰部分に触覚刺激を感じるタイミングがユーザBよりも遅れてしまう場合がある。これは、振動部52Cからの触覚刺激がユーザの左右の腰部分に到達するまでに時間を要するためである。
これにより、異なるタイプの触覚信号再生機器32A,32Bを装着したユーザに対して、左右の腰部分に同じタイミングで触覚刺激を提供することができる。
例えば、振動部52の種類によっては、再生可能な振動強度が異なる場合について考える。再生可能な振動強度が低い振動部52であるにも関わらず大きな振動を発生させてしまうと、それよりも大きな振動を発生させるべきシーンが到来した際にその大きな振動を発生させることができなくなってしまう虞がある。その結果、振動の強弱による表現の豊かさを失ってしまい、適切な触覚刺激をユーザに体感させることができなくなってしまう。
例えば、ある振動部52から出力された振動における高域の減衰が大きい場合には、高域のゲインを上げることにより対処してもよい。
調整処理の第4例においては、ユーザとスクリーン24の距離である距離情報に応じた調整処理を行う例を説明する。
このような距離情報に応じた調整処理は、ユーザの状態情報と合わせて行うことが望ましい。
提示制御装置31の通信制御部46は、ユーザに対して各種の通知を行うための送信制御を行うことが可能である。ここではいくつかの通知例を説明する。
通信制御部46は、ユーザに提供する触覚刺激が基準触覚信号に基づくものではなく、調整後の調整触覚信号に基づくものであることを通知する処理を行う。この処理は、調整処理部45による調整処理が行われたことに応じて実行される。
このような通知を受け取ることにより、ユーザは最適な調整がなされた触覚刺激を体感していることを認識することができると共に、基準姿勢とは異なる姿勢となっていることを認識することができる。
同様に、通知処理においては、通知内容を示すデータがペイロードに格納されて送信することができる。但し、このような態様では、通知処理のためのパケットデータを生成して送信することが必要となってしまう。
従って、複数のパケットデータに亘って通知処理に係る送信データを格納することにより、効率よくデータの送信を行うことが可能となる。
なお、このような方法で制御情報や通知情報を受信した触覚信号再生機器32は、複数のパケットデータから制御情報や通知情報を復元する処理を実行することにより、受信データを復元することができる。
上述した提示制御装置31や触覚信号再生機器32は、上述した各機能を備えるために、コンピュータ装置としての構成を備えている。なお、各装置や機器が以下に示す全ての構成を備えている必要はなく、一部のみを備えていてもよい。
各機器のCPU71は、図21に示すように、ROM72や例えばEEP-ROM(Electrically Erasable Programmable Read-Only Memory)などの不揮発性メモリ部74に記憶されているプログラム、または記憶部79からRAM73にロードされたプログラムに従って各種の処理を実行する。RAM73にはまた、CPU71が各種の処理を実行する上において必要なデータなども適宜記憶される。
CPU71、ROM72、RAM73、不揮発性メモリ部74は、バス83を介して相互に接続されている。このバス83にはまた、入出力インタフェース75も接続されている。
例えば入力部76としては、キーボード、マウス、キー、ダイヤル、タッチパネル、タッチパッド、リモートコントローラ等の各種の操作子や操作デバイスが想定される。
入力部76によりユーザの操作が検知され、入力された操作に応じた信号はCPU71によって解釈される。
表示部77は各種表示を行う表示部であり、例えばコンピュータ装置の筐体に設けられるディスプレイデバイスや、コンピュータ装置に接続される別体のディスプレイデバイス等により構成される。
表示部77は、CPU71の指示に基づいて表示画面上に各種の画像処理のための画像や処理対象の動画等の表示を実行する。また表示部77はCPU71の指示に基づいて、各種操作メニュー、アイコン、メッセージ等、即ちGUI(Graphical User Interface)としての表示を行う。
ドライブ81により、リムーバブル記憶媒体82からは画像ファイル等のデータファイルや、各種のコンピュータプログラムなどを読み出すことができる。読み出されたデータファイルは記憶部79に記憶されたり、データファイルに含まれる画像や音声が表示部77や音声出力部78で出力されたりする。またリムーバブル記憶媒体82から読み出されたコンピュータプログラム等は必要に応じて記憶部79にインストールされる。
なお、提示制御装置31や触覚信号再生機器32を構成する情報処理装置は、図21のようなコンピュータ装置が単一で構成されることに限らず、複数のコンピュータ装置がシステム化されて構成されてもよい。複数のコンピュータ装置は、LAN等によりシステム化されていてもよいし、インターネット等を利用したVPN等により遠隔地に配置されたものでもよい。複数の情報処理装置には、クラウドコンピューティングサービスによって利用可能なサーバ群(クラウド)としての情報処理装置が含まれてもよい。
上述の各種の例を実現するために提示制御装置31のCPU71が実行する処理について添付図を参照して説明する。
一例を図24に示す。
情報取得処理では、先ず提示制御装置31のCPU71はステップS201において、デバイス情報を取得する。デバイス情報は、触覚信号再生機器32が備える振動部52の個数や配置位置や周波数特性などの情報を取得する処理である。
一方、基準デバイスと異なると判定した場合、提示制御装置31のCPU71はステップS103において、デバイス情報に応じた調整処理を実行する。例えば、振動部52が二つ設けられている基準デバイスに対して、振動部52が一つしか設けられていない触覚信号再生機器32を着用しているユーザに対しては、基準触覚信号の二つの信号を合成した上で信号強度を増加させた調整触覚信号を生成する。
例えば、基準触覚信号の制作者が調整処理を許諾していない場合や、触覚信号再生機器32を着用しているユーザが調整処理を許可していない場合には、提示制御装置31のCPU71はステップS105,ステップS106及びステップS107の各処理を実行せずに図23のステップS108へと進む。
一方、ユーザの状態が基準状態と異なっている場合には、提示制御装置31のCPU71はステップS106において、状態情報に応じた調整処理を行い、ステップS107において調整が行われたことをユーザに通知する通知処理を行う。
上述した例では、提示制御装置31において基準触覚信号の調整処理を行い調整触覚信号を生成する例を説明したが、調整処理を触覚信号再生機器32で行ってもよい。
また、図27に示すように、触覚信号再生機器32は判定処理部43と識別情報設定部44と調整処理部45と再生制御部51と振動部52と姿勢検出部53と通信制御部54を備えている。
触覚信号再生機器32は、姿勢検出部53によって検出されたユーザの姿勢情報や、振動部52が備える感圧素子などによって検出された情報などを取得する。更に、触覚信号再生機器32は、取得した各種情報に基づいて調整処理を行うか否かを判定する。調整処理を行う場合には、調整処理部45は、提示制御装置31から取得した基準触覚信号に調整処理を施し、調整触覚信号を生成する。そして、再生制御部51は、基準触覚信号や調整触覚信号を振動部52に出力することにより触覚刺激の提示を実現する。
例えば、ゲーム機と表示装置とゲーム用コントローラを備えたゲームシステムにおいて、上述した提示制御装置31をゲーム機に置き換え、上述した触覚信号再生機器32をゲーム用コントローラに置き換えることにより、より臨場感を増したゲームコンテンツを楽しむことが可能である。
上述したように、本技術の信号処理装置としての提示制御装置31は、ユーザの姿勢を含むユーザの状態を示す状態情報を取得する情報取得部41と、基準状態のユーザに対する触覚提示を行うための基準触覚信号を取得する触覚信号取得部42と、状態情報に基づいて基準触覚信号を調整することにより調整触覚信号を生成する調整処理部45と、を備えている。
ユーザの姿勢が異なれば、ユーザが体感する触覚刺激も異なる。本構成によれば、姿勢を含めたユーザの状態を示す状態情報に基づいて基準触覚信号を調整することにより、ユーザの状態に合わせた調整触覚信号が生成される。
これにより、適切な触覚信号に基づく触覚提示をユーザに対して行うことができる。従って、臨場感や没入感をより増すことができ、ユーザ体験の向上を図ることができる。
このような信号処理装置としての提示制御装置31は、例えば、ユーザの腕などに装着する触覚信号再生機器32などに好適である。ユーザの腕の位置や向きなどは、ユーザ毎に異なる可能性が高く、通常であればユーザの腕の向きや位置に応じた適切な触覚提示を行うことができる場面は非常に限られている。しかし、そのような触覚信号再生機器32について上記した構成を適用することにより、ユーザが腕をどのような姿勢にしていたとしても、意図した触覚提示を適切にユーザの腕に提供することができる。これにより、臨場感や没入感を高めることができる。
ユーザにとって適切な触覚刺激は、ユーザの姿勢等に応じて異なる場合と、ユーザの姿勢によらず同一である場合がある。本構成によれば、ユーザの状態情報に基づく調整を行う場合と行わない場合とを適宜切り換えることが可能となる。
従って、姿勢に応じて調整を行いたい触覚信号のみを対象として調整を行うことなどが可能となる。
触覚信号の調整を行うか否かについては、触覚信号の種別等に応じて自動で決定するだけでなくユーザが決定することも考えられる。
本構成によれば、ユーザ自身が触覚信号の調整を行うか否かを決定することができるため、ユーザの好みなどに適応することが可能となる。
これにより、基準触覚信号を調整すべきか否かについてユーザの姿勢を加味して適切に判定することができる。
従って、ユーザの姿勢に応じて適切な触覚刺激を与えることができる。また、ユーザの姿勢と基準状態が多少異なっていたとしても調整は不要と判定されることもあり、そのような場合には、調整処理の実行が回避されるため、提示制御装置31の処理負担の軽減が図られる。
調整が不能とされた場合に触覚信号の再生が行われないようにすることで、ユーザに触覚刺激を与えないようにすることができる。
従って、却って臨場感や没入感を失わせてしまうような不適切な触覚刺激が再生されてしまう可能性を排除することができる。
これにより、触覚信号の送信処理の処理負担が軽減される。
従って、複数のユーザが使用するそれぞれの触覚信号再生機器32(32A,32B,32C,32D,32E)に触覚信号を送信する場合などに、処理のタスクがオーバーフローしてしまい送信処理が適切なタイミングで実行されなくなってしまうことを回避することができる。また、受信側である触覚信号再生機器32の受信処理や再生処理に係る処理負担についても軽減される。
これにより、触覚信号の調整を行う信号処理装置としての提示制御装置31は、調整の要否によらず一律触覚信号の送信処理を行うことができる。例えば、複数の触覚信号再生機器に対して触覚信号を送信する場合に、基準状態と大きく異なる姿勢をとるユーザが装着する特定の触覚信号にだけ触覚信号を送信しないように制御する必要がない。
そして、触覚信号の再生を実行するかしないかについては、触覚信号を受信した触覚信号再生機器32で行われる。従って、信号処理装置の処理負担を軽減することができる。
ユーザは、各種の状況で触覚刺激を体感することが考えられる。そして、その中には、コンテンツに集中している場合と集中していない場合がある。例えば、映像と共に触覚刺激を楽しむユーザが、コンテンツに集中しているか否かを判定する手法としては、映像をきちんと視聴しているか否かを判定することが考えられる。
画面に集中していないユーザは、別のことをしながらコンテンツを楽しんでいることが考えられる。本構成によれば、そのようなユーザに対して触覚刺激の提示を中断もしくは中止することができ、ユーザがコンテンツの視聴と共に行っている別の行動を妨げないようにすることができる。具体的には、例えば、字を書きながらコンテンツを見ているようなユーザは、画面を見ていない瞬間が多々ある。そのようなユーザに触覚刺激を提供してしまうと、字が正確に書けないなどユーザの行動の妨げになってしまう虞があるが、このようなユーザを適切に判定して触覚刺激の提示を中断もしくは中止することにより、ユーザが複数のことを同時進行で行うことの妨げになってしまうことを防止することができる。
これにより、ユーザは、基準触覚信号が調整されていることを把握することができる。
従って、自分の姿勢や体勢を考慮して触覚刺激の意味を認識しなおさずに済むことを把握することができる。即ち、触覚刺激を素直に体感することで制作者の意図した触覚刺激が受けられることを自覚することができるため、臨場感や没入感を味わうことができる。
これにより、ユーザの状態に応じて適切な強度の触覚提示を行うことが可能となる。
これにより、ユーザの状態の違いだけでなく装着している触覚信号再生機器32の違いに応じて適切な調整がなされた触覚信号を送信することが可能となる。
従って、ユーザは、自身の姿勢や装着している触覚信号再生機器32などの違いを意識することなく、触覚信号再生機器32によって提供される触覚刺激を体感するだけで、最適なユーザ体験を得ることができる。
これにより、振動部52の配置位置が異なる触覚信号再生機器32を各ユーザが着用していたとしても、ユーザごとに適切な触覚信号が生成される。
従って、ユーザは、姿勢の違いだけでなく振動部52の配置位置の違いを意識することなく触覚刺激を享受することができる。
また、振動部52の位置によってはユーザが振動を体感する際の強度が異なる場合がある。本構成では、振動部52の位置に応じてユーザが感じる振動の強さの違いを考慮した調整を行うことが可能となるため、制作者の意図をより忠実に汲み取った触覚刺激をユーザに提供することができるため、より高い臨場感や没入感をユーザが体験することが可能となる。
ユーザが映像を見ながら触覚刺激を楽しむ場合において、映像の見え方が異なれば、最適な触覚刺激も異なる可能性が高い。
本構成によれば、ユーザが表示装置に近いほど、触覚提示の強度を増すことなどが実現可能であるため、ユーザが感じる臨場感や没入感を更に高めることができる。
ユーザの姿勢が変化すれば、最適な触覚刺激も異なる。
姿勢の変化に応じてユーザの最新の状態情報を取得することにより、ユーザが断続的に動いているような状況であっても最適な触覚刺激を提供し続けることができる。
また、上述した各例は、組み合わせが不可能でない限りいかように組み合わせることも可能である。
本技術は以下のような構成を採ることもできる。
(1)
ユーザの姿勢を含むユーザの状態を示す状態情報を取得する情報取得部と、
基準状態のユーザに対する触覚提示を行うための基準触覚信号を取得する触覚信号取得部と、
前記状態情報に基づいて前記基準触覚信号を調整することにより調整触覚信号を生成する調整処理部と、を備えた
信号処理装置。
(2)
前記調整の実行可否を判定する判定処理部を備えた
上記(1)に記載の信号処理装置。
(3)
前記判定処理部は、ユーザの操作に基づいて前記実行可否の判定結果を決定する
上記(2)に記載の信号処理装置。
(4)
前記判定処理部は、ユーザの姿勢と前記基準状態が異なることによって前記調整が必要となるか否かを判定する
上記(2)から上記(3)の何れかに記載の信号処理装置。
(5)
触覚信号の再生を行う触覚信号再生機器に触覚信号を送信する送信制御を行う通信制御部を備え、
前記通信制御部は、前記調整が不能であると判定された場合に触覚信号の再生が前記触覚信号再生機器で行われないように送信制御を行う
上記(4)に記載の信号処理装置。
(6)
前記通信制御部は、前記調整が不能であると判定された場合に触覚信号の送信を行わない
上記(5)に記載の信号処理装置。
(7)
前記通信制御部は、前記調整が不能であると判定された場合に触覚信号の再生が行われないように制御情報を付加して触覚信号の送信を行う
上記(5)に記載の信号処理装置。
(8)
触覚信号の再生を行う触覚信号再生機器に触覚信号を送信する送信制御を行う通信制御部を備え、
前記情報取得部は、ユーザに視認させる映像をユーザが視認しているか否かを示す視認情報を取得し、
前記通信制御部は、ユーザが前記映像を視認していないことを示す情報が前記視認情報として取得された場合に、触覚信号の再生が前記触覚信号再生機器で行われないように送信制御を行う
上記(1)から上記(7)の何れかに記載の信号処理装置。
(9)
触覚信号の再生を行う触覚信号再生機器に触覚信号を送信する送信制御を行う通信制御部を備え、
前記通信制御部は、前記調整触覚信号を送信する場合に調整を行ったことを通知するための送信制御を行う
上記(1)から上記(8)の何れかに記載の信号処理装置。
(10)
前記調整処理部は、前記基準触覚信号の調整において前記触覚提示の強度を変更する調整を行う
上記(1)から上記(9)の何れかに記載の信号処理装置。
(11)
前記調整処理部は、ユーザが装着している触覚信号再生機器に応じて前記調整を行う
上記(10)に記載の信号処理装置。
(12)
前記調整処理部は、前記触覚信号再生機器が備える触覚提示部の位置と前記状態情報に応じて前記触覚提示の強度及びタイミングを調整する
上記(11)に記載の信号処理装置。
(13)
前記情報取得部は、ユーザに視認させる映像が表示される表示装置とユーザの距離情報を取得し、
前記調整処理部は、取得された前記距離情報に応じて前記触覚提示の強度を変更する調整を行う
上記(10)から上記(12)の何れかに記載の信号処理装置。
(14)
前記情報取得部は、ユーザの前記姿勢が変化するごとに前記状態情報を取得する
上記(1)から上記(13)の何れかに記載の信号処理装置。
(15)
ユーザの姿勢を含むユーザの状態を示す状態情報を取得する処理と、
基準状態のユーザに対する触覚提示を行うための基準触覚信号を取得する処理と、
前記状態情報に基づいて前記基準触覚信号を調整することにより調整触覚信号を生成する処理と、を信号処理装置が実行する
信号処理方法。
(16)
ユーザの姿勢を含むユーザの状態を示す状態情報を取得する情報取得部と、基準状態のユーザに対する触覚提示を行うための基準触覚信号を取得する触覚信号取得部と、前記状態情報に基づいて前記基準触覚信号を調整することにより調整触覚信号を生成する調整処理部と、を備えた信号処理装置と、
前記調整触覚信号の再生を行う触覚信号再生機器と、を備えた
信号処理システム。
24 スクリーン(表示装置)
31 提示制御装置(信号処理装置)
32,32A,32B,32C,32D,32E 触覚信号再生機器
41 情報取得部
42 触覚信号取得部
43 判定処理部
44 識別情報設定部
45 調整処理部
46 通信制御部
52,52C,52R,52L 振動部(触覚提示部)
Claims (16)
- ユーザの姿勢を含むユーザの状態を示す状態情報を取得する情報取得部と、
基準状態のユーザに対する触覚提示を行うための基準触覚信号を取得する触覚信号取得部と、
前記状態情報に基づいて前記基準触覚信号を調整することにより調整触覚信号を生成する調整処理部と、を備えた
信号処理装置。 - 前記調整の実行可否を判定する判定処理部を備えた
請求項1に記載の信号処理装置。 - 前記判定処理部は、ユーザの操作に基づいて前記実行可否の判定結果を決定する
請求項2に記載の信号処理装置。 - 前記判定処理部は、ユーザの姿勢と前記基準状態が異なることによって前記調整が必要となるか否かを判定する
請求項2に記載の信号処理装置。 - 触覚信号の再生を行う触覚信号再生機器に触覚信号を送信する送信制御を行う通信制御部を備え、
前記通信制御部は、前記調整が不能であると判定された場合に触覚信号の再生が前記触覚信号再生機器で行われないように送信制御を行う
請求項4に記載の信号処理装置。 - 前記通信制御部は、前記調整が不能であると判定された場合に触覚信号の送信を行わない
請求項5に記載の信号処理装置。 - 前記通信制御部は、前記調整が不能であると判定された場合に触覚信号の再生が行われないように制御情報を付加して触覚信号の送信を行う
請求項5に記載の信号処理装置。 - 触覚信号の再生を行う触覚信号再生機器に触覚信号を送信する送信制御を行う通信制御部を備え、
前記情報取得部は、ユーザに視認させる映像をユーザが視認しているか否かを示す視認情報を取得し、
前記通信制御部は、ユーザが前記映像を視認していないことを示す情報が前記視認情報として取得された場合に、触覚信号の再生が前記触覚信号再生機器で行われないように送信制御を行う
請求項1に記載の信号処理装置。 - 触覚信号の再生を行う触覚信号再生機器に触覚信号を送信する送信制御を行う通信制御部を備え、
前記通信制御部は、前記調整触覚信号を送信する場合に調整を行ったことを通知するための送信制御を行う
請求項1に記載の信号処理装置。 - 前記調整処理部は、前記基準触覚信号の調整において前記触覚提示の強度を変更する調整を行う
請求項1に記載の信号処理装置。 - 前記調整処理部は、ユーザが装着している触覚信号再生機器に応じて前記調整を行う
請求項10に記載の信号処理装置。 - 前記調整処理部は、前記触覚信号再生機器が備える触覚提示部の位置と前記状態情報に応じて前記触覚提示の強度及びタイミングを調整する
請求項11に記載の信号処理装置。 - 前記情報取得部は、ユーザに視認させる映像が表示される表示装置とユーザの距離情報を取得し、
前記調整処理部は、取得された前記距離情報に応じて前記触覚提示の強度を変更する調整を行う
請求項10に記載の信号処理装置。 - 前記情報取得部は、ユーザの前記姿勢が変化するごとに前記状態情報を取得する
請求項1に記載の信号処理装置。 - ユーザの姿勢を含むユーザの状態を示す状態情報を取得する処理と、
基準状態のユーザに対する触覚提示を行うための基準触覚信号を取得する処理と、
前記状態情報に基づいて前記基準触覚信号を調整することにより調整触覚信号を生成する処理と、を信号処理装置が実行する
信号処理方法。 - ユーザの姿勢を含むユーザの状態を示す状態情報を取得する情報取得部と、基準状態のユーザに対する触覚提示を行うための基準触覚信号を取得する触覚信号取得部と、前記状態情報に基づいて前記基準触覚信号を調整することにより調整触覚信号を生成する調整処理部と、を備えた信号処理装置と、
前記調整触覚信号の再生を行う触覚信号再生機器と、を備えた
信号処理システム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202180063759.2A CN116209973A (zh) | 2020-11-10 | 2021-10-08 | 信号处理装置、信号处理方法和信号处理*** |
US18/251,538 US20240012482A1 (en) | 2020-11-10 | 2021-10-08 | Signal processing apparatus, signal processing method, and signal processing system |
DE112021005923.8T DE112021005923T5 (de) | 2020-11-10 | 2021-10-08 | Signalverarbeitungsvorrichtung, signalverarbeitungsverfahren und signalverarbeitungssystem |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2020-187387 | 2020-11-10 | ||
JP2020187387 | 2020-11-10 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022102307A1 true WO2022102307A1 (ja) | 2022-05-19 |
Family
ID=81601125
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2021/037381 WO2022102307A1 (ja) | 2020-11-10 | 2021-10-08 | 信号処理装置、信号処理方法、信号処理システム |
Country Status (4)
Country | Link |
---|---|
US (1) | US20240012482A1 (ja) |
CN (1) | CN116209973A (ja) |
DE (1) | DE112021005923T5 (ja) |
WO (1) | WO2022102307A1 (ja) |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2018109974A (ja) * | 2016-12-27 | 2018-07-12 | イマージョン コーポレーションImmersion Corporation | 視野を用いる触覚フィードバック |
WO2020026443A1 (ja) * | 2018-08-03 | 2020-02-06 | 株式会社ソニー・インタラクティブエンタテインメント | 触覚表現用振動制御システム、触覚表現用振動発生装置、触覚表現用振動制御装置、および触覚表現用振動制御方法 |
WO2020054415A1 (ja) * | 2018-09-11 | 2020-03-19 | ソニー株式会社 | 情報処理装置、情報処理方法、及び、記録媒体 |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150323994A1 (en) | 2014-05-07 | 2015-11-12 | Immersion Corporation | Dynamic haptic effect modification |
-
2021
- 2021-10-08 DE DE112021005923.8T patent/DE112021005923T5/de active Pending
- 2021-10-08 US US18/251,538 patent/US20240012482A1/en active Pending
- 2021-10-08 WO PCT/JP2021/037381 patent/WO2022102307A1/ja active Application Filing
- 2021-10-08 CN CN202180063759.2A patent/CN116209973A/zh active Pending
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2018109974A (ja) * | 2016-12-27 | 2018-07-12 | イマージョン コーポレーションImmersion Corporation | 視野を用いる触覚フィードバック |
WO2020026443A1 (ja) * | 2018-08-03 | 2020-02-06 | 株式会社ソニー・インタラクティブエンタテインメント | 触覚表現用振動制御システム、触覚表現用振動発生装置、触覚表現用振動制御装置、および触覚表現用振動制御方法 |
WO2020054415A1 (ja) * | 2018-09-11 | 2020-03-19 | ソニー株式会社 | 情報処理装置、情報処理方法、及び、記録媒体 |
Also Published As
Publication number | Publication date |
---|---|
CN116209973A (zh) | 2023-06-02 |
DE112021005923T5 (de) | 2023-09-28 |
US20240012482A1 (en) | 2024-01-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10993498B2 (en) | Vibrating footwear device and entertainment system for use therewith | |
EP3253468B1 (en) | Motion sickness monitoring and application of supplemental sound to counteract sickness | |
JP6835070B2 (ja) | システム、端末装置、方法及び記録媒体 | |
US11120674B2 (en) | Information processing apparatus, information processing method, and program | |
JP2017182130A (ja) | 情報処理装置、情報処理方法、及びプログラム | |
WO2009103823A2 (en) | Method and system for fitting hearing systems | |
US20220122333A1 (en) | Information processing apparatus, information processing method, and program | |
WO2022102307A1 (ja) | 信号処理装置、信号処理方法、信号処理システム | |
JP2018191159A (ja) | 動画の配信方法 | |
US11190874B2 (en) | Information processing device and information processing method | |
US11228855B2 (en) | Information processing device and information processing method | |
JP2009201800A (ja) | 運動支援装置、コンピュータプログラム及び運動支援方法 | |
US20210409876A1 (en) | Method for Adjusting a Hearing Aid Device and System for Carrying Out the Method | |
JP7172999B2 (ja) | 情報処理装置、情報処理方法及びプログラム | |
JP2021094086A (ja) | ゲームシステム及びプログラム | |
WO2024101001A1 (en) | Information processing system, information processing method, and program for communication points regarding events | |
JP2021171320A (ja) | 身体状態再現方法、及び、身体状態再現システム | |
WO2022190919A1 (ja) | 情報処理装置、情報処理方法、およびプログラム | |
JP2022173870A (ja) | 鑑賞システム、鑑賞装置及びプログラム | |
CN113412138A (zh) | 姿势改善辅助装置、姿势改善辅助方法以及姿势改善辅助程序 | |
JP2019114930A (ja) | 情報処理装置及びプログラム |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21891554 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 18251538 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 112021005923 Country of ref document: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21891554 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: JP |