CN115758107B - Haptic signal transmission method and device, storage medium and electronic equipment - Google Patents
Haptic signal transmission method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN115758107B CN115758107B CN202211338523.8A CN202211338523A CN115758107B CN 115758107 B CN115758107 B CN 115758107B CN 202211338523 A CN202211338523 A CN 202211338523A CN 115758107 B CN115758107 B CN 115758107B
- Authority
- CN
- China
- Prior art keywords
- signal
- signals
- characteristic
- audio
- haptic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 230000008054 signal transmission Effects 0.000 title claims abstract description 16
- 238000000605 extraction Methods 0.000 claims abstract description 35
- 238000003062 neural network model Methods 0.000 claims abstract description 31
- 239000000463 material Substances 0.000 claims abstract description 29
- 238000007906 compression Methods 0.000 claims abstract description 28
- 230000006835 compression Effects 0.000 claims abstract description 28
- 230000008447 perception Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 5
- 230000002068 genetic effect Effects 0.000 claims description 4
- 210000002569 neuron Anatomy 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 24
- 230000005540 biological transmission Effects 0.000 abstract description 11
- 230000008569 process Effects 0.000 description 8
- 230000024159 perception of rate of movement Effects 0.000 description 7
- 238000003672 processing method Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000004088 simulation Methods 0.000 description 6
- 238000001514 detection method Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 239000000284 extract Substances 0.000 description 4
- 238000009432 framing Methods 0.000 description 4
- 230000004927 fusion Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000035807 sensation Effects 0.000 description 3
- 239000002023 wood Substances 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003155 kinesthetic effect Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000007654 immersion Methods 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000035772 mutation Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000704 physical effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- User Interface Of Digital Computer (AREA)
Abstract
The disclosure provides a haptic signal transmission method and device, a storage medium and electronic equipment, and relates to the technical field of signal processing. Collecting touch signals, image data and audio/video data; determining the material type according to the image data, and obtaining force feedback characteristics under the corresponding material; inputting the haptic signal, the force feedback characteristic and the audio/video data into a preset neural network model for characteristic extraction, and obtaining a multi-mode input signal; compression encoding is carried out on the multimode input signal based on Weber's law, so as to obtain a multimode characteristic signal; and sending the multi-mode characteristic signal to a receiving end so that the receiving end simulates according to the multi-mode characteristic signal. Effective compression is realized, the data transmission quantity is reduced, and the accuracy of the representation of the touch signal is improved.
Description
Technical Field
The disclosure relates to the technical field of signal processing, and in particular relates to a method and a device for transmitting a touch signal, a storage medium and electronic equipment.
Background
With the rapid development of digital signal processing technology and communication technology, virtual reality technology is coming into the field of view of the public, and the way of blending virtual world and real world is an important direction of technological development and research. From audiovisual interactions to multi-sensory interactions, multimedia interaction experiences began to incorporate haptic sensations in order to bring more extreme interactive sensations and richer experiences to the user. The importance of haptic sensation as a perceived manner next to auditory and visual is self-evident.
However, the techniques such as haptic signal compression and reconstruction are still in the starting and exploring stage, and many problems exist. At present, the traditional tactile signal has different degrees of signal delay or loss due to oversized data packets in the signal transmission process, so that the problems of disordered tactile signal time sequence, poor action simulation experience and the like can be caused.
Therefore, there is a need to solve the problems of delay and loss of the signal caused by the oversized data packet during the transmission of the haptic signal.
It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The disclosure aims to provide a haptic signal transmission method and device, a storage medium and an electronic device, which at least overcome the problem of oversized data packets in the transmission process in the related art to a certain extent.
Other features and advantages of the present disclosure will be apparent from the following detailed description, or may be learned in part by the practice of the disclosure.
According to one aspect of the present disclosure, there is provided a method for processing a multi-mode signal, applied to a transmitting end, including:
collecting touch signals, image data and audio/video data;
determining the material type according to the image data, and obtaining force feedback characteristics under the corresponding material;
inputting the haptic signal, the force feedback characteristic and the audio/video data into a preset neural network model for characteristic extraction, and obtaining a multi-mode input signal;
compression encoding is carried out on the multimode input signal based on Weber's law, so as to obtain a multimode characteristic signal;
and sending the multi-mode characteristic signal to a receiving end so that the receiving end simulates according to the multi-mode characteristic signal.
In one embodiment of the present disclosure, the multi-modal input signal includes: a haptic input signal, an audio-visual input signal, and image information;
inputting the haptic signal, the force feedback feature and the audio/video data into a preset neural network model for feature extraction, and obtaining a multi-mode input signal, wherein the method comprises the following steps:
acquiring characteristic information of the force feedback characteristics and label information of the audio and video data;
and extracting signal characteristics of the corresponding tactile signals in the same time domain according to the characteristic information and the label information to obtain a tactile input signal.
In one embodiment of the present disclosure, before the step of inputting the haptic signal, the force feedback feature and the audio/video data into a preset neural network model to perform feature extraction, the method includes:
detecting a signal type corresponding to the audio and video data;
and performing cross-mode signal coding according to the signal type to obtain an audio and video input signal.
In one embodiment of the present disclosure, the haptic signal includes: vibration signal, friction signal and pressure signal.
In one embodiment of the present disclosure, the multi-modal signature signal includes: haptic feature signals, audio-visual feature signals, and image feature information;
the compression encoding is performed on the multi-mode input signal based on weber's law to obtain a multi-mode characteristic signal, which comprises the following steps:
determining a perception blind area range of the tactile input signal based on the Weber's law;
removing signals of the tactile input signals in the sensing blind area range according to the sensing blind area range to obtain tactile characteristic signals;
and acquiring image feature information and audio and video feature signals corresponding to the time domain according to the time domain of the touch feature signal.
In one embodiment of the disclosure, the preset neural network model includes an input layer, a hidden layer, and an output layer; the number of neurons of the input and output layers is determined based on the dimension of the haptic characteristic signal.
In one embodiment of the present disclosure, the method includes:
and determining the initialization parameters of the preset neural network model by using a genetic algorithm.
According to another aspect of the present disclosure, there is provided a processing apparatus for multi-modal signals, including:
the information acquisition module is used for acquiring the touch signal, the image data and the audio/video data;
the force characteristic extraction module is used for determining the material type according to the image data and obtaining force feedback characteristics under the corresponding material;
the feature extraction module is used for inputting the tactile signals, the force feedback features and the audio/video data into a preset neural network model to perform feature extraction, and obtaining multi-mode input signals;
the compression coding module is used for carrying out compression coding on the multi-mode input signal based on Weber's law to obtain a multi-mode characteristic signal;
and the signal sending module is used for sending the multi-mode characteristic signal to a receiving end so that the receiving end simulates according to the multi-mode characteristic signal.
According to still another aspect of the present disclosure, there is provided an electronic apparatus including:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the method of processing a multi-modal signal of any one of the above via execution of the executable instructions.
According to yet another aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of processing a multi-modal signal of any one of the above.
The embodiment of the disclosure provides a method for processing a multi-mode signal, which is applied to a transmitting end and comprises the following steps: collecting touch signals, image data and audio/video data; determining the material type according to the image data, and obtaining force feedback characteristics under the corresponding material; inputting the haptic signal, the force feedback characteristic and the audio/video data into a preset neural network model for characteristic extraction, and obtaining a multi-mode input signal; compression encoding is carried out on the multimode input signal based on Weber's law, so as to obtain a multimode characteristic signal; and sending the multi-mode characteristic signal to a receiving end so that the receiving end simulates according to the multi-mode characteristic signal. The haptic signals of the actual scene are restored with the assistance of the image and audio/video information, and the accuracy of the haptic signals is improved; and the multimode input signal is effectively compressed through the pre-characteristic extraction and compression coding process, so that the data transmission quantity is reduced, and the accuracy of the representation of the touch signal is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. It will be apparent to those of ordinary skill in the art that the drawings in the following description are merely examples of the disclosure and that other drawings may be derived from them without undue effort.
FIG. 1 illustrates a prior art haptic signal transmission schematic diagram in one embodiment of the present disclosure;
FIG. 2 is a flow chart of a method for processing a multi-mode signal according to one embodiment of the disclosure;
FIG. 3 illustrates a multi-modal signal flow diagram in one embodiment of the present disclosure;
FIG. 4 is a flow chart of a multi-mode signal processing method according to another embodiment of the disclosure;
FIG. 5 is a flow chart of a multi-modal signal processing method in yet another embodiment of the present disclosure;
FIG. 6 is a flow chart of a multi-mode signal processing method according to another embodiment of the disclosure;
FIG. 7 is a schematic diagram of a multi-mode signal processing device according to an embodiment of the disclosure; and
fig. 8 shows a block diagram of an electronic device in one embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.
The scheme provided by the application, namely the compression and reconstruction of the touch signal, is one of core technologies for realizing deep immersion and interactive experience.
In the practical application scene, the transmission and the receiving of signals usually have different degrees of loss, so that the modal signals are not complete. In the prior related art scheme, a conventional haptic signal transmission scheme is shown in fig. 1. The pressure of the tactile signal is collected by the pressure collecting unit 110, the collected pressure is processed by the framing unit 120 to form a frame by a plurality of rows and columns of pressure, the framing information is then transmitted to the information receiving unit 140 of the receiving end by the information transmitting unit 130 through the communication network, the received framing information is then de-framed by the de-framing unit 150, and the controlled unit 170 is controlled by the control unit 160 to realize signal simulation according to the de-framed information.
In the prior related scheme, only the frames formed by the pressure of a plurality of rows and a plurality of columns are used for characterizing the touch sense, and the characterization is single. And there is still a problem that there are different degrees of delay or loss due to how large the data packet is in the signaling process.
Accordingly, to solve the above-mentioned problems, embodiments of the present disclosure provide a multi-mode signal processing method. The following description refers to the accompanying drawings.
In an embodiment of the present disclosure, a method for processing a multi-mode signal is provided, which is applied to a transmitting end and includes:
s201, collecting touch signals, image data, audio and video data;
specifically, as shown in a multi-mode signal flow diagram in fig. 3, information is collected by three different modules respectively, and tactile signals are collected by the tactile collection unit 301 respectively, where the tactile signals are derived from the pressure of the touching object on the skin, and include the surface shape, temperature, friction force, etc. of the object. The image detecting unit 302 acquires image information in the current scene, for example, a touch object in the current scene such as wood, metal, etc., and then acquires the audio and video information in the current scene, for example, acquires the currently emitted clicking sound and the corresponding video picture, through the audio and video acquisition unit 303.
S202, determining the material type according to the image data, and obtaining force feedback characteristics under the corresponding material;
referring to fig. 3, in the image detection unit 302, a material type may be determined according to the image data, for example, it is determined that the material of the object that is currently acting is wood according to the image data, and the material type is encoded, and then the characteristic of the acting force corresponding to the wood material is determined. Optionally, the step can train the neural network model to learn the force feedback characteristics under different materials so as to achieve the monitoring purpose. The above steps are not only from the perspective of tactile pressure, but also can comprehensively and accurately assist in restoring the tactile signal by combining the material type of the current application scene.
S203, inputting the haptic signal, the force feedback characteristic and the audio/video data into a preset neural network model for characteristic extraction, and obtaining a multi-mode input signal;
specifically, in the process of feature extraction, the feature extraction unit 304 inputs the haptic signal, the feedback feature of force and the audio/video data in a unified scene into the preset neural network model, extracts the haptic signal in the current scene, extracts the representation signal feature through deep learning, mainly extracts the feature of multiple types of haptic signals such as key friction and vibration, extracts the relatively complete signal, and can better represent the haptic information, and then obtains the multi-mode input signal, so as to restore the haptic signal to a greater extent. The multi-mode input signals comprise tactile input signals, material types, feedback characteristics of force and audio/video input signals after feature extraction.
S204, carrying out compression coding on the multi-mode input signal based on Weber' S law to obtain a multi-mode characteristic signal;
specifically, during the compression encoding process, based on weber-fishena's law of cognitive deviation in the compression encoding module 305, only when the current kinesthesia signal is significantly different from the previous kinesthesia signal, people can feel the same, and the kinesthesia data can be compressed by adjusting the area size of the blind area under the condition that the user experience is not affected. And removing the signals which are trapped in human perception dead zones due to the small stimulus in a short time, and realizing the compression coding of human perception interval signals. The method can effectively compress the data, reduce the data transmission quantity and reduce the delay and the deletion degree in the data transmission process. After compression encoding, a multi-modal characteristic signal is obtained, the multi-modal characteristic signal specifically comprising: the information of the time sequence characteristic labels and the like corresponding to the compressed touch input signals and the compressed audio and video input signals, and the information of different dimensions such as material types and the like.
And S205, the multi-mode characteristic signal is sent to a receiving end, so that the receiving end simulates according to the multi-mode characteristic signal.
Specifically, referring to fig. 3, the compressed multi-mode feature signal is sent to the receiving end through the information sending module 306, the information receiving module 307 of the receiving end decodes the multi-mode feature signal through the decoding unit 308, and then performs fusion simulation through the multi-mode fusion unit 309 after decoding, in the fusion simulation process, the multi-mode feature signal is received, fusion is performed based on the acquired signals with different features, then the control unit 310 controls the controlled unit 311 to perform simulation, and the user senses physical properties such as friction and vibration of the object through simulating the object with the same touch with the signal parameters, so that the interactive experience of the user is improved.
The processing method of the multi-mode signal provided in the embodiment collects the touch signal, the image data and the audio/video data; determining the material type according to the image data, and obtaining force feedback characteristics under the corresponding material; inputting the haptic signals, the force feedback characteristics and the audio/video data into a preset neural network model for characteristic extraction, and obtaining multi-mode input signals; compression encoding is carried out on the multimode input signal based on Weber's law, so as to obtain a multimode characteristic signal; and sending the multi-mode characteristic signal to a receiving end so that the receiving end simulates according to the multi-mode characteristic signal. The haptic signals of the actual scene are restored with the assistance of the image and audio/video information, and the accuracy of the haptic signals is improved; the multi-mode input signal is effectively compressed through the pre-characteristic extraction and compression coding process, so that the data transmission quantity is reduced, and the accuracy of the representation of the touch signal is improved.
In an embodiment of the present disclosure, the multi-modal input signal includes: a haptic input signal, an audio-visual input signal, and image information;
as shown in fig. 4, a flowchart of another multi-mode signal processing method, the inputting the haptic signal, the force feedback feature and the audio/video data into a preset neural network model for feature extraction, to obtain a multi-mode input signal, includes:
s401, acquiring characteristic information of the force feedback characteristic and label information of the audio and video data;
s402, extracting signal characteristics of the corresponding tactile signals in the same time domain according to the characteristic information and the label information, and obtaining tactile input signals.
Specifically, the preset neural network model is trained in advance, the test set and the training set are determined by using the collected output signal data, the preset neural network model is initialized at first, the training is performed after the initialization, and the trained preset neural network model can obtain relatively complete touch input signals through the auxiliary extraction of the characteristic information of the collected force feedback characteristics and the label information of the audio-video data. The method is characterized in that the characteristics of the key friction, vibration and other multi-type tactile signals are extracted, relatively complete signals are extracted, and the tactile information can be better represented.
In the characteristic extraction process, the abstract label information in the image and audio/video signals is utilized to combine the tactile signals with the object material image, audio/video signals of the same scene, and the tactile signals in the same scene are subjected to characteristic extraction to restore the tactile signals to the greatest extent.
In another embodiment of the present disclosure, before the step of inputting the haptic signal, the force feedback feature and the audio/video data into a preset neural network model to perform feature extraction, the method further includes:
s501, detecting a signal type corresponding to the audio and video data;
s502, performing cross-mode signal coding according to the signal type to obtain an audio and video input signal.
Specifically, audio and video information when the touch scene occurs is collected, the signal type is detected, and cross-modal signal encoding is carried out on the signal type. The encoded audio/video input signal. The detection signal types can be the detection of the sound and the picture of the audio and video data in the current application scene, and the detection signal types respectively correspond to different signal types. Wherein, the cross-modal coding is to use the semantic relativity between multi-modal code streams to carry out joint coding, and the steps realize the cross-modal coding of the audio and the video.
In an embodiment of the present disclosure, the haptic signal includes: vibration signal, friction signal and pressure signal.
Optionally, the haptic signals in this embodiment include, but are not limited to, vibration signals, friction signals, and pressure signals, temperature signals, etc. The type of the haptic signal can be determined according to the actual application scene.
As shown in fig. 6, in another flow chart of the multi-mode signal processing method, in an embodiment of the present disclosure, the multi-mode characteristic signal includes: haptic feature signals, audio-visual feature signals, and image feature information;
the compression encoding is performed on the multi-mode input signal based on weber's law to obtain a multi-mode characteristic signal, which comprises the following steps:
s601, determining a perception blind area range of a touch input signal based on the Weber law;
s602, eliminating signals of the tactile input signals in the sensing blind area range according to the sensing blind area range to obtain tactile characteristic signals;
and S603, acquiring image feature information and audio and video feature signals corresponding to the time domain according to the time domain of the touch feature signal.
Specifically, only when the current kinesthesia signal is significantly different from the previous kinesthesia signal, people can feel the current kinesthesia signal, and the kinesthesia data can be compressed under the condition that the user experience is not influenced by adjusting the size of the area of the perception blind area. And removing the signals which are trapped in human perception dead zones due to the small stimulus in a short time, and realizing the compression coding of human perception interval signals. For example, m is a threshold value of a range of a perception dead zone, an input sampling point is a kinesthetic signal X, and a signal at a time T adjacent to X is the kinesthetic signal X T According to the following expression:
IX T -XI≤m·X T (1)
the signal change at time T is not perceived by a person because of small amplitude, and therefore, can be discarded from transmission. Discarding the perceptually dead zone signal can greatly reduce the transmission rate of the data packet.
And selecting image feature information and audio/video feature signals corresponding to the time domain according to the determined touch feature signals, and ensuring that the signals are in the same scene.
In an embodiment of the present disclosure, the preset neural network model includes an input layer, a hidden layer, and an output layer; the number of neurons of the input and output layers is determined based on the dimension of the haptic characteristic signal.
In the above-mentioned preset neural network model, in the model training process, the neural network model needs to have an input layer, a hidden layer and an output layer. The number of neurons in the input and output layers is selected correspondingly by the number of haptic signal input and output dimensions.
In an embodiment of the present disclosure, the method includes:
and determining the initialization parameters of the preset neural network model by using a genetic algorithm.
Specifically, in the model initialization process, the weight and the threshold of the neural network are utilized to form population individuals of a genetic algorithm, then the population individuals are encoded, after the fitness function is selected, the intersection and mutation operation is carried out, the individual with the highest fitness is repeatedly searched, and the individual is used as the initial value and the threshold of the optimal neural network and is then used for training the neural network model. The feature extraction capability of the obtained preset neural network model is more accurate and efficient.
In another embodiment of the present disclosure, there is provided a multi-mode signal processing apparatus, as shown in fig. 7, comprising:
an information acquisition module 701 for acquiring haptic signals, image data, audio and video data;
a force characteristic extraction module 702, configured to determine a material type according to the image data, and obtain a force feedback characteristic under a corresponding material;
the feature extraction module 703 is configured to input the haptic signal, the force feedback feature and the audio/video data into a preset neural network model for feature extraction, so as to obtain a multi-mode input signal;
the compression coding module 704 is configured to perform compression coding on the multi-mode input signal based on weber's law to obtain a multi-mode characteristic signal;
the signal sending module 705 is configured to send the multi-mode feature signal to a receiving end, so that the receiving end performs simulation according to the multi-mode feature signal.
The present embodiment provides a processing device for multi-mode signals, which comprises an information acquisition module 701, a force feature extraction module 702, a feature extraction module 703, a compression coding module 704, and a signal transmission module 705. The haptic signals of the actual scene are restored with the assistance of the image and audio/video information, and the accuracy of the haptic signals is improved; the multimode input signals are effectively compressed through the pre-characteristic extraction module and the compression coding module, so that the data transmission quantity is reduced, and the accuracy of the representation of the touch signals is improved.
In yet another embodiment of the present disclosure, there is provided an electronic device including:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the method of processing a multi-modal signal of any one of the above via execution of the executable instructions.
The electronic device provided in this embodiment realizes the method for processing the multi-mode signal through the processor.
And will not be described in detail herein.
In yet another embodiment of the present disclosure, a computer readable storage medium is provided, on which a computer program is stored, which when executed by a processor, implements the method for processing a multi-modal signal according to any one of the above.
The embodiment provides a computer readable storage medium, which is executed by a processor through a computer program to implement the above-mentioned method for processing a multi-mode signal.
And will not be described in detail herein.
An electronic device 800 according to such an embodiment of the application is described below with reference to fig. 8. The electronic device 800 shown in fig. 8 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present application.
As shown in fig. 8, the electronic device 800 is embodied in the form of a general purpose computing device. Components of electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one memory unit 820, and a bus 830 connecting the various system components, including the memory unit 820 and the processing unit 810.
Wherein the storage unit stores program code that is executable by the processing unit 810 such that the processing unit 810 performs steps according to various exemplary embodiments of the present application described in the above section of the "exemplary method" of the present specification. For example, the processing unit 810 may perform S201 acquisition of haptic signals, image data, and audio-video data as shown in fig. 2; s202, determining the material type according to the image data, and obtaining force feedback characteristics under the corresponding material; s203, inputting the haptic signals, the force feedback characteristics and the audio/video data into a preset neural network model for characteristic extraction, and obtaining multi-mode input signals; s204, carrying out compression coding on the multi-mode input signal based on Weber' S law to obtain a multi-mode characteristic signal; s205, the multi-mode characteristic signal is sent to the receiving end, so that the receiving end simulates according to the multi-mode characteristic signal.
The storage unit 820 may include readable media in the form of volatile storage units, such as Random Access Memory (RAM) 8201 and/or cache memory 8202, and may further include Read Only Memory (ROM) 8203.
Storage unit 820 may also include a program/utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
Bus 830 may be one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and/or any device (e.g., router, modem, etc.) that enables the electronic device 800 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 850. Also, electronic device 800 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 over bus 830. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 800, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.
In an exemplary embodiment of the present disclosure, a computer-readable storage medium having stored thereon a program product capable of implementing the method described above in the present specification is also provided. In some possible embodiments, the various aspects of the application may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the application as described in the "exemplary methods" section of this specification, when said program product is run on the terminal device.
A program product for implementing the above method according to an embodiment of the present application is described, which may employ a portable compact disc read-only memory (CD-ROM) and comprise program code and may be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.
Furthermore, although the steps of the methods in the present disclosure are depicted in a particular order in the drawings, this does not require or imply that the steps must be performed in that particular order or that all illustrated steps be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform, etc.
From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
Claims (9)
1. A method for transmitting a haptic signal, applied to a transmitting end, comprising:
collecting touch signals, image data and audio/video data;
determining the material type according to the image data, and obtaining force feedback characteristics under the corresponding material;
inputting the haptic signal, the force feedback characteristic and the audio/video data into a preset neural network model for characteristic extraction, and obtaining a multi-mode input signal;
compression encoding is carried out on the multimode input signal based on Weber's law, so as to obtain a multimode characteristic signal;
the multi-mode characteristic signals are sent to a receiving end, so that the receiving end simulates according to the multi-mode characteristic signals;
wherein the multi-modal input signal comprises: a haptic input signal, an audio-visual input signal, and image information;
inputting the haptic signal, the force feedback feature and the audio/video data into a preset neural network model for feature extraction, and obtaining a multi-mode input signal, wherein the method comprises the following steps:
acquiring characteristic information of the force feedback characteristics and label information of the audio and video data;
and extracting signal characteristics of the corresponding tactile signals in the same time domain according to the characteristic information and the label information to obtain a tactile input signal.
2. A haptic signal transmission method as recited in claim 1 wherein prior to said step of inputting said haptic signal, said force feedback characteristics and said audio-visual data into a predetermined neural network model for feature extraction, obtaining a multi-modal input signal comprises:
detecting a signal type corresponding to the audio and video data;
and performing cross-mode signal coding according to the signal type to obtain an audio and video input signal.
3. A haptic signal transmission method as recited in claim 1 wherein said haptic signal includes: vibration signal, friction signal and pressure signal.
4. A haptic signal transmission method as recited in claim 1 wherein said multi-modal feature signal includes: haptic feature signals, audio-visual feature signals, and image feature information;
the compression encoding is performed on the multi-mode input signal based on weber's law to obtain a multi-mode characteristic signal, which comprises the following steps:
determining a perception blind area range of a touch input signal based on the weber law;
removing signals of the tactile input signals in the sensing blind area range according to the sensing blind area range to obtain tactile characteristic signals;
and acquiring image feature information and audio and video feature signals corresponding to the time domain according to the time domain of the touch feature signal.
5. A haptic signal transmission method as recited in claim 4 wherein said predetermined neural network model includes an input layer, a hidden layer, and an output layer; the number of neurons of the input and output layers is determined based on the dimension of the haptic characteristic signal.
6. A haptic signal transmission method as recited in claim 5 wherein said method includes:
and determining the initialization parameters of the preset neural network model by using a genetic algorithm.
7. A haptic signal transmission device, comprising:
the information acquisition module is used for acquiring the touch signal, the image data and the audio/video data;
the force characteristic extraction module is used for determining the material type according to the image data and obtaining force feedback characteristics under the corresponding material;
the feature extraction module is used for inputting the tactile signals, the force feedback features and the audio/video data into a preset neural network model to perform feature extraction, and obtaining multi-mode input signals; wherein the multi-modal input signal comprises: a haptic input signal, an audio-visual input signal, and image information;
the compression coding module is used for carrying out compression coding on the multi-mode input signal based on Weber's law to obtain a multi-mode characteristic signal;
the signal sending module is used for sending the multi-mode characteristic signals to a receiving end so that the receiving end simulates the multi-mode characteristic signals;
the feature extraction module comprises: the information acquisition module is used for acquiring the characteristic information of the force feedback characteristic and the label information of the audio and video data;
and the signal acquisition module is used for extracting signal characteristics of the corresponding tactile signals in the same time domain according to the characteristic information and the label information to obtain tactile input signals.
8. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the haptic signal transmission method of any one of claims 1-6 via execution of the executable instructions.
9. A computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the haptic signal transmission method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211338523.8A CN115758107B (en) | 2022-10-28 | 2022-10-28 | Haptic signal transmission method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211338523.8A CN115758107B (en) | 2022-10-28 | 2022-10-28 | Haptic signal transmission method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115758107A CN115758107A (en) | 2023-03-07 |
| CN115758107B true CN115758107B (en) | 2023-11-14 |
Family
ID=85355957
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211338523.8A Active CN115758107B (en) | 2022-10-28 | 2022-10-28 | Haptic signal transmission method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115758107B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104049749A (en) * | 2013-03-15 | 2014-09-17 | 英默森公司 | Method and apparatus to generate haptic feedback from video content analysis |
| CN109447038A (en) * | 2018-11-27 | 2019-03-08 | 南京恩诺网络科技有限公司 | Tactile data evaluation method and device, system |
| CN109559758A (en) * | 2018-11-05 | 2019-04-02 | 清华大学 | A method of texture image is converted by haptic signal based on deep learning |
| CN111190481A (en) * | 2018-11-14 | 2020-05-22 | 意美森公司 | Systems and methods for generating haptic effects based on visual characteristics |
| CN113840588A (en) * | 2021-08-15 | 2021-12-24 | 曹庆恒 | Touch sensing system and use method thereof |
| CN114225379A (en) * | 2021-12-20 | 2022-03-25 | 努比亚技术有限公司 | Game scene vibration regulation and control method, equipment and computer readable storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9274603B2 (en) * | 2013-05-24 | 2016-03-01 | Immersion Corporation | Method and apparatus to provide haptic feedback based on media content and one or more external parameters |
-
2022
- 2022-10-28 CN CN202211338523.8A patent/CN115758107B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104049749A (en) * | 2013-03-15 | 2014-09-17 | 英默森公司 | Method and apparatus to generate haptic feedback from video content analysis |
| CN109559758A (en) * | 2018-11-05 | 2019-04-02 | 清华大学 | A method of texture image is converted by haptic signal based on deep learning |
| CN111190481A (en) * | 2018-11-14 | 2020-05-22 | 意美森公司 | Systems and methods for generating haptic effects based on visual characteristics |
| CN109447038A (en) * | 2018-11-27 | 2019-03-08 | 南京恩诺网络科技有限公司 | Tactile data evaluation method and device, system |
| CN113840588A (en) * | 2021-08-15 | 2021-12-24 | 曹庆恒 | Touch sensing system and use method thereof |
| CN114225379A (en) * | 2021-12-20 | 2022-03-25 | 努比亚技术有限公司 | Game scene vibration regulation and control method, equipment and computer readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 基于人眼感知特性的亮度系数压缩方法;喻莉;郭姗;徐士麟;周刚;李荣;;中国图象图形学报(03);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115758107A (en) | 2023-03-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP4012706A1 (en) | Training method and device for audio separation network, audio separation method and device, and medium | |
| CN111933115B (en) | Speech recognition method, apparatus, device and storage medium | |
| CN110929098B (en) | Video data processing method and device, electronic equipment and storage medium | |
| CN105578115A (en) | Network teaching method and system with voice assessment function | |
| CN111711834B (en) | Recorded broadcast interactive course generation method and device, storage medium and terminal | |
| CN112423081B (en) | Video data processing method, device and equipment and readable storage medium | |
| CN113254684B (en) | Content aging determination method, related device, equipment and storage medium | |
| US10600337B2 (en) | Intelligent content parsing with synthetic speech and tangible braille production | |
| CN112399258A (en) | Live playback video generation playing method and device, storage medium and electronic equipment | |
| CN110516749A (en) | Model training method, method for processing video frequency, device, medium and calculating equipment | |
| CN113793398A (en) | Drawing method and device based on voice interaction, storage medium and electronic equipment | |
| CN112214591A (en) | Conversation prediction method and device | |
| CN115019237B (en) | Multi-mode emotion analysis method and device, electronic equipment and storage medium | |
| CN109166409B (en) | Sign language conversion method and device | |
| CN113392273A (en) | Video playing method and device, computer equipment and storage medium | |
| WO2023197749A9 (en) | Background music insertion time point determining method and apparatus, device, and storage medium | |
| CN113411674A (en) | Video playing control method and device, electronic equipment and storage medium | |
| CN111144138A (en) | Simultaneous interpretation method and device and storage medium | |
| CN115758107B (en) | Haptic signal transmission method and device, storage medium and electronic equipment | |
| CN110852801B (en) | Information processing method, device and equipment | |
| Wojtanowski et al. | “Alexa, Can You See Me?” Making Individual Personal Assistants for the Home Accessible to Deaf Consumers | |
| CN115223244B (en) | Haptic motion simulation method, device, apparatus and storage medium | |
| CN115116444A (en) | Processing method, device and equipment for speech recognition text and storage medium | |
| CN113571063A (en) | Voice signal recognition method and device, electronic equipment and storage medium | |
| CN111858856A (en) | Multi-round search type chatting method and display equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |