KR20220145570A - Method and apparatus for tactile sensing - Google Patents

Abstract

A tactile sensing device is disclosed. The device inputs a time-series touch signal and sliding signal obtained from a memory for storing a pre-learned tactile simulation model and a detection target into the tactile simulation model, and tactile information corresponding to a tactile level mapped to a predetermined user or user group. It includes a control unit that provides As the device is provided, user convenience can be provided.

Description

Tactile sensing method and device

The present disclosure relates to a method and apparatus for simulating a user's tactile sense and sensing a tactile sense, and a recording medium storing a program for performing the tactile sensing method.

Digital experiences based on the five senses continue to evolve thanks to advances in electronic device manufacturing technology and signal processing technology. VR (Virtual Reality) technology has caused visual and auditory sensations, and personalized services that provide three-dimensional images and stereo sound are being provided through VR technology and AR (Augmented Reality) technology.

For an immersive digital experience, it is key to accurately detect touch and effectively deliver it. Unlike sight and hearing, there were technical difficulties in detecting, expressing, and delivering touch.

In addition, there is a technical difficulty in imitating the touch of a specific user even if the sense of touch is simply sensed because there is individual difference in the sense of touch for each user. Accordingly, there is a need for a method for sensing a tactile sense in a user-customized manner.

On the other hand, the above information is only presented as background information to help the understanding of the present invention. No decision has been made, nor is any claim made as to whether any of the above is applicable as prior art to the present invention.

Korean Patent Application Laid-Open No. 10-2018-0099283 (published on September 5, 2018)

SUMMARY OF THE INVENTION An object of the present invention is to provide an apparatus and method for sensing a touch by simulating a user-specific touch.

Another object of the present invention is to provide an apparatus and method for providing tactile information corresponding to a tactile level mapped to a user or a user group using a neural network algorithm.

Another object of the present invention is to provide an apparatus and method for providing tactile information corresponding to a user to a tactile reproduction apparatus.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. will be able

In order to achieve the above object, a tactile sensing apparatus according to an embodiment of the present invention includes: a memory for storing a pre-learned tactile simulation model; and a control unit that inputs a time-series touch signal and a sliding signal obtained from a detection target into the tactile simulation model, and provides tactile information corresponding to a tactile level mapped to a predetermined user or user group. have.

A tactile sensing method performed by a processor according to an embodiment of the present invention includes: inputting a time-series touch signal and a sliding signal obtained from a sensing target into a pre-trained tactile simulation model; and providing tactile information corresponding to a tactile level mapped to a predetermined user or user group.

The tactile sensing method according to an embodiment of the present invention may be stored in a computer-readable recording medium in which a program to be executed on a computer is recorded.

When the program is executed by a processor, the processor includes: an operation of inputting, by the processor, a time-series touch signal and a sliding signal obtained from a detection target into a pre-learned tactile simulation model; and an executable command for outputting tactile information corresponding to a tactile level mapped to a predetermined user or user group.

The means of solving the technical problems to be achieved in the present invention are not limited to the solutions mentioned above, and other solutions not mentioned are clear to those of ordinary skill in the art to which the present invention belongs from the description below. will be able to be understood

According to various embodiments of the present disclosure, an apparatus for sensing tactile sense by simulating a tactile sense for each user is provided, thereby improving user convenience and performing an accurate tactile sensing process for each user.

The effects of the present invention are not limited to the effects of the invention mentioned above, and the effects of other inventions not mentioned above can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. There will be.

1 is a diagram schematically illustrating an apparatus for sensing a tactile sense from a sensing target according to an embodiment of the present invention;
2 is a block diagram showing the configuration of a tactile sensing device according to an embodiment of the present invention;
3 is a block diagram showing the configuration of a tactile sensing device equipped with a tactile sensor according to an embodiment of the present invention;
4 is a diagram for explaining label information input to a tactile simulation model in a learning step according to an embodiment of the present invention;
5 shows an algorithm for explaining the operation of a tactile simulation model according to an embodiment of the present invention;
6 (a) to 6 (c) are diagrams for explaining the tactile determination of a plurality of participants according to various embodiments of the present invention;
7 is a view for explaining a change in accuracy according to a change in the tolerance range according to an embodiment of the present invention;
8 is a graph comparing a user and a tactile determination of a tactile sensing device corresponding to the user before and after learning according to an embodiment of the present invention;
9 is a sequence diagram illustrating a tactile sensing method according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings. The present invention may be embodied in a variety of different forms, and should not be construed as being limited to the embodiments described herein. The relative sizes of components, layers, and regions in the drawings may be exaggerated for clarity of description.

1 is a diagram schematically illustrating an apparatus (Ro 1, Ro N) for sensing a tactile sense from a sensing target according to an embodiment of the present invention.

The tactile sensing devices (Ro 1, Ro N) may correspond to each user (USER 1, USER N), and by simulating the tactile characteristics of each user (USER 1, USER N), the tactile sense from the sensing target (Ta, Target) information can be extracted.

The tactile sensing devices Ro 1 and Ro N may be implemented as a user-customized avatar robot, and the configuration and function of the first tactile sensing device Ro1 will be schematically described.

The first tactile sensing device Ro 1 extracts tactile information of the detection target Ta by using the tactile sensor Se 1 disposed on the robot arm Ar 1 , and the extracted tactile information is transmitted to the first user ( It may be tactile information corresponding to the tactile level of USER 1).

Here, the tactile sensor Se 1 may be disposed by forming a layer of one or more tactile sensors. In particular, the tactile sensor Se 1 may include a tactile sensor array composed of columns and rows. The first tactile sensing device Ro 1 may drive the robot arm Ar 1 to vertically and/or horizontally move the robot arm Ar 1 .

The first tactile sensing device Ro 1 may measure the degree of hardness (hardness) of the sensing target Ta through a touch input with respect to the sensing target Ta, and may measure the sensing target Ta through a sliding input. ) can measure the degree of roughness (surface topography).

2 is a block diagram showing the configuration of the tactile sensing apparatus 100 (100A) according to an embodiment of the present invention.

The tactile sensing apparatuses 100 and 100A may be implemented as a robot as shown in FIG. 1 , and may be implemented as a device that mainly performs calculations without including a tactile sensing robot. The tactile sensing apparatuses 100 and 100A of FIG. 2 are assumed to be devices that mainly perform calculations and will be representatively described.

The tactile sensing apparatus 100 may include a memory 150 and a control unit 190 that performs an operation.

First, the memory 150 may be implemented using various semiconductors, and may store a pre-trained tactile simulation model TM. The tactile simulation model (TM) may be a model in which learning has been completed, or may be a model in the learning process. In an optional embodiment, the memory 150 may store only the data of the tactile simulation model TM necessary for performing the necessary operation, and additional data stored in the external device may be received at a necessary time and used for the operation.

The tactile simulation model TM may be a neural network-based model already trained to output tactile information corresponding to a tactile level mapped to a predetermined user or user group.

Here, the tactile level mapped to the user is a tactile level (tactile sense level) that simulates the user's tactile sense at a predetermined level (for example, within a 5% similarity range), and the tactile sense level mapped to the user group is the same within a predetermined range. Alternatively, it may be a tactile level (tactile sense level) of a group of users having a similar tactile level.

In addition, the tactile information may be determined hardness information and roughness information of a specific detection target, and may include probability distribution information expressing them. The probability distribution information may be expressed as a histogram, and may indicate a degree of smoothness and a degree of roughness of a specific detection target.

The controller 190 inputs the time-series touch signal and the sliding signal obtained from the detection target to the tactile simulation model TM to provide tactile information corresponding to the tactile level mapped to a predetermined user or user group. can

3 is a block diagram illustrating the configuration of the tactile sensing apparatuses 100 and 100B on which the tactile sensor is mounted according to an embodiment of the present invention. The tactile sensing devices 100 and 100B illustrated in FIG. 3 may be devices that not only perform calculations but also directly collect tactile information as a sensing target.

The tactile sensing apparatus 100 may include a tactile sensor 110 , a robot arm 120 , a display 130 , a memory 150 storing a tactile simulation model TM , and a controller 190 .

The tactile sensor 110 is a sensor that senses a tactile sense, and may be disposed at a predetermined position of the robot arm 120 . In an embodiment, the tactile sensor 110 may be disposed in a hand region of the robot arm 120 .

The robot arm 120 (Arm) is a module capable of physically moving the tactile sensor 110 , and may be driven according to a control signal of the controller 190 .

The controller 190 may control the robot arm 120 so that the tactile sensor 110 applies pressure to the detection target to obtain a time-series touch signal. The time-series touch signal may be a signal related to a time-series voltage value obtained by measuring the hardness of the sensing target, that is, hardness.

Also, the controller 190 may control the robot arm 120 to obtain a sliding signal by horizontally moving the tactile sensor 110 while being touched by the detection target. That is, the sliding signal may be a signal for the degree of bumpiness, ie, surface roughness, measured while the tactile sensor 110 horizontally moves the surface of the detection target in a touched state according to a predetermined speed and a predetermined intensity.

However, when inputting a sliding signal to the tactile simulation model TM, the controller 190 performs Fourier transform on the obtained voltage value, not the voltage value obtained while moving in the horizontal direction (value in the frequency domain). ) can be entered. This is to acquire only a signal of a vibration frequency range (eg, within 1 KHz) that can be perceived by human touch, to achieve a complex, and to more effectively acquire roughness information.

The tactile sensing apparatus 100 may include a display 130 . The display 130 is a module capable of outputting various information, and the controller 190 may output information and tactile information about a predetermined user or user group through the display 130 .

The memory 150 stores the simulated tactile model TM as described above. The simulated tactile model TM uses tactile information corresponding to a tactile level mapped to a predetermined user or user group as label information. can be trained

4 is a diagram for explaining label information input to a tactile simulation model (TM) in a learning step according to an embodiment of the present invention.

Referring to FIG. 4 , in the detection targets N1 and N2 to NZ, the first detection target N1 is the softest and the Z-th detection target NZ is the roughest. A plurality of users sense the tactile sense of the detection targets N1 to NZ, and each user may touch the first detection target N1 to the Z-th detection target NZ.

Here, touching the first detection target N1 to the Z-th detection target NZ once may be referred to as one trial. The user can repeat (1T~NT) for N trials. The first sensing target N1 to the Nth sensing target NZ may be an LMT haptic texture, but in an optional embodiment, they may be implemented with other materials.

In an optional embodiment, the user may sequentially touch the detection target from the first detection target N1 to the Z-th detection target NZ at a predetermined speed and a predetermined intensity. Alternatively, the user may touch the first detection target N1 to the Z-th detection target NZ in a random order.

The user may perform the N touch test on a specific detection target (eg, N3 ), and according to the same or different results for each trial, this may be displayed as a histogram. The histogram may represent a tactile level for a specific detection target (eg, N3 ) of a specific user (or user group) as a probability distribution.

The tactile sensing apparatus 100 may train (learn) a tactile simulation model TM mapped to a user (or user group) to output tactile information corresponding to a tactile level for each user (or user group).

That is, the tactile sensing apparatus 100 may simulate the tactile level of each user (or user group) to simulate the tactile ability of the corresponding user. Accordingly, since user-customized tactile sensing can be performed, user convenience can be improved, and tactile reliability can be improved.

5 shows an algorithm for explaining the operation of the tactile simulation model (TM) according to an embodiment of the present invention.

The tactile simulation model TM may be a supervised learning-based neural network model, and may include a first subnetwork 1S, a second subnetwork 2S, and a third subnetwork 3S.

First, the first sub-network 1S may be a sub-network for receiving a time-series touch signal (eg, a voltage value) to obtain a voltage gradient of the time-series touch signal. The first sub-network 1S is a neural network-based network using a property that the hardness value increases as the voltage value of the touch signal increases. That is, the higher the voltage value of the touch signal, the harder the material, and the first subnetwork 1S may determine this.

The first subnetwork 1S may include a one-dimensional convolution layer and a max pooling layer. Since the first subnetwork 1S only needs to check the slope of the voltage value, it includes only a simple structure in feature extraction. . Here, the max pulling layer is a layer for efficiently obtaining a peak voltage value.

The second subnetwork 2S is a subnetwork for extracting surface roughness information of a detection target based on an input frequency signal, and may be configured as a deep neural network (DNN).

When receiving a sliding signal, the second subnetwork 2S sets the measured voltage value so as to more easily distinguish the rubbing input, solve the complexity, and distinguish the range of vibration frequencies that can be sensed by the human touch. Values can be extracted by applying a Fourier transform that transforms into the frequency domain. In addition, the extracted sliding signal may be input to the second subnetwork 2S.

The third subnetwork 3S receives the output value of the first subnetwork 1S and the output value of the second subnetwork 2S, and can integrate the first subnetwork 1S and the second subnetwork 2S. have.

The third sub-network 3S is a sub-network for more precisely discriminating tactile information corresponding to a tactile level mapped to a predetermined user or user group. It may include a fully connected neural network of ?

The tactile simulation model TM is to output tactile information corresponding to a tactile level mapped to a predetermined user or user group, so that the parameters of the first sub-network 1S to the third sub-network 3S are updated at each repetition interval. can The parameter is a parameter of the CNN algorithm or the DNN algorithm, and may be updated at every iteration, and may be updated until the last iteration (EPOC).

The tactile simulation model TM may provide tactile information corresponding to a tactile level mapped to a user (a level having a specific sensing characteristic with respect to a specific sensing target). The tactile information may be provided based on a probability distribution.

In addition, the tactile sensing apparatus 100 may provide tactile information corresponding to the tactile level mapped to the output user (or user group) to the tactile reproducing apparatus, and the tactile reproducing apparatus outputs the corresponding tactile information. information and vibration frequency information.

In the tactile simulation model TM, when the tactile information corresponding to the user's tactile level and the tactile sensing device 100 simulates the user and outputs the tactile information, the comparison tolerance of the two tactile information can be set, and the tolerance is met. Thus, the tactile information of the tactile sensing apparatus 100 may be set to follow the user's tactile information, and related parameters may be repeatedly updated.

6A to 6C are diagrams for explaining a tactile determination of a plurality of participants according to various embodiments of the present disclosure. Specifically, FIG. 6(a) shows the average of tactile information provided by each of a plurality of users who have sensed a plurality of detection targets according to an embodiment of the present invention, and FIG. 6(b) is an embodiment of the present invention. It is a graph showing the RMSE of each of a plurality of participants together with respect to the root mean square deviation (RMSE) of the tactile determination of all the participants according to the present invention. It is a graph representing

In FIG. 6( a ) , each of a plurality of participants may perform a tactile sensing test on a plurality of (eg, 42 types) detection targets based on a multi-trial. A plurality of participants may experiment with how many detection targets with different tactile information are set as detection targets. For example, assuming that the first participant decides the detection target with the 25th hardness and roughness from the 23rd to the 27th, if the Tolerance is set to 2, this is the correct answer, but if the Tolerance is set to 1, the detection date is out of range. can In addition, overlapping scores may be distinguished by a color difference on the graph.

는 촉각 결정 정보이며, i 는 l 번째까지 수행하는 실험(Trial) 인덱스이며, s는 참가자의 수(S)이며, m은 감지 타깃의 수(M)이다. k는 최대 K이며, m 또는 s 나타낼 수 있다.Referring to FIG. 6(b) , the RMSE for the tactile determination may be derived by [Equation 1] below. here,

is the tactile determination information, i is the trial index performed up to the lth, s is the number of participants (S), and m is the number of detection targets (M). k is at most K, and can be represented by m or s.

[Equation 1]

Referring to FIG. 6(b) , it can be divided into S group and D group based on the participant's average tactile determination RMSE. The S group may be a group occupying the majority of users, and the D group may be a group having a difference from the S group within a predetermined range.

Referring to FIG. 6C , standard deviations of tactile determination information of the S group and the D group may be calculated differently. In general, a similar pattern will be seen on a rough or smooth sensing target, but the pattern may deviate from a sensing target that may be rough or soft. That is, there can be large variance in tactile determination in materials that are not too rough or too soft, where multiple participants may confuse roughness information.

7 is a view for explaining a change in accuracy according to a change in a tolerance range according to an embodiment of the present invention.

Referring to FIG. 7 , as the range of tolerance is set wider, the accuracy of the user's tactile determination, the accuracy of the label information, and the accuracy of the histogram may be set higher.

In addition, it can be observed that the accuracy of the user's tactile determination, the accuracy of the label information, and the accuracy of the histogram converge similarly as the range of tolerance is set wider.

8 is a graph comparing tactile determinations of a user and a tactile sensing device corresponding to the user before and after learning according to an embodiment of the present invention.

는 촉감 감지 장치(100)가 제공하는 촉감 정보(촉감 결정 정보)이다.First, the RMSE representing the difference between the tactile information (tactile determination information) of a plurality of participants and the tactile information (tactile determination information) of the tactile sensing device can be derived as shown in [Equation 2] below. here,

is tactile information (tactile determination information) provided by the tactile sensing apparatus 100 .

[Equation 2]

Referring to FIG. 8 , the RMSE for the tactile determination of the size material and the intermediate material before learning is high, but the RMSE for the tactile determination of the size material and the intermediate material after learning may be measured lower than this.

That is, when the pre-learned tactile simulation model TM according to an embodiment of the present invention is applied, tactile information (tactile determination information) may be more accurate.

9 is a sequence diagram illustrating a tactile sensing method according to an embodiment of the present invention.

The tactile sensing method may be performed by the controller 190 of the tactile sensing apparatus 100 , and the controller 190 may include one or more processors.

In the tactile sensing method, a time-series touch signal and a sliding signal obtained from a sensing target are input to a pre-learned tactile simulation model (S910).

Next, the tactile sensing method provides tactile information corresponding to a tactile level mapped to a predetermined user or user group (S920).

Before step S910, the tactile sensing method applies pressure to the sensing target using a tactile sensor to obtain a time-series touch signal (S901). After that, in the tactile sensing method, the tactile sensor moves horizontally while being touched by the sensing target to obtain a sliding signal (S903). Steps S901 and S903 may be performed simultaneously or in a different order.

The present invention described above can be implemented as computer-readable code on a medium in which a program is recorded. The computer-readable medium includes all kinds of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Also, the computer may include the control unit 190 of the tactile sensing apparatus 100 .

In the foregoing, specific embodiments of the present invention have been described and illustrated, but the present invention is not limited to the described embodiments, and those of ordinary skill in the art may make various changes to other specific embodiments without departing from the spirit and scope of the present invention. It will be appreciated that modifications and variations are possible. Accordingly, the scope of the present invention should not be defined by the described embodiments, but should be defined by the technical idea described in the claims.

Claims (18)

A tactile sensing device comprising:
a memory for storing the pre-trained tactile simulation model; and
Tactile sensing, comprising a control unit that inputs a time-series touch signal and a sliding signal obtained from a detection target into the tactile simulation model and provides tactile information corresponding to a tactile level mapped to a predetermined user or user group; Device. According to claim 1,
tactile sensor; and
The tactile sensor is disposed at a predetermined position, further comprising a robot arm (Arm) for physically moving the tactile sensor,
The control unit is
Control the robot arm so that the tactile sensor applies pressure to the detection target to obtain a time-series touch signal, and controls the robot arm to obtain a sliding signal by horizontally moving the tactile sensor while being touched on the detection target Consisting of a tactile sensing device. According to claim 1,
The time-series touch signal is related to a time-series voltage value for measuring the hardness of the sensing target,
The sliding signal is
In order to extract the surface roughness information of the sensing target, the measured voltage value is Fourier transformed into a frequency signal and input to the tactile simulation model. According to claim 1,
The tactile simulation model is,
A tactile sensing device, which is trained by using tactile information corresponding to a tactile level mapped to a predetermined user or user group as label information. According to claim 1,
The tactile simulation model is a supervised learning-based neural network model, and includes first to third subnetworks,
an output value of the first subnetwork and an output value of the second subnetwork are used as an input value of the third subnetwork;
The tactile sensing apparatus, wherein parameters of the first to third subnetworks are updated for each repetition interval so as to output tactile information corresponding to a tactile level mapped to a predetermined user or user group. 6. The method of claim 5,
The first subnetwork is
A tactile sensing device, which is a subnetwork for obtaining a voltage gradient of a time-series touch signal, and includes a one-dimensional convolutional layer and a max pooling layer. 7. The method of claim 6,
The second subnetwork,
A tactile sensing device, comprising: a sub-network for extracting surface roughness information of a sensing target based on an input frequency signal, and configured of a deep neural network (DNN). 8. The method of claim 7,
The third subnetwork,
a sub-network integrating the first sub-network and the second sub-network;
A tactile sensing apparatus, which is a sub-network for more precisely classifying tactile information corresponding to a tactile level mapped to a predetermined user or user group. According to claim 1,
further comprising a display,
The tactile information output by the tactile simulation model includes a probability distribution indicating tactile information corresponding to a tactile level mapped to a predetermined user or user group,
The control unit is
and outputting information on the predetermined user or user group and the tactile information through the display. A tactile sensing method performed by a processor, comprising:
inputting a time-series touch signal and a sliding signal obtained from a detection target into a pre-trained tactile simulation model; and
A tactile sensing method comprising the step of providing tactile information corresponding to a tactile level mapped to a predetermined user or user group. 11. The method of claim 10,
Before the input step,
obtaining a time-series touch signal by applying pressure to a detection target using a tactile sensor; and
The tactile sensing method further comprising the step of obtaining a sliding signal by horizontally moving the tactile sensor while being touched by the detection target. 11. The method of claim 10,
The time-series touch signal is related to a time-series voltage value for measuring the hardness of the sensing target,
The sliding signal is
In order to extract the surface roughness information of the sensing target, the measured voltage value is Fourier transformed into a frequency signal and input to the tactile simulation model. 11. The method of claim 10,
The tactile simulation model is,
A tactile sensing method, which is trained by using tactile information corresponding to a tactile level mapped to a predetermined user or user group as label information. 11. The method of claim 10,
The tactile simulation model is a supervised learning-based neural network model, and includes first to third subnetworks,
an output value of the first subnetwork and an output value of the second subnetwork are used as an input value of the third subnetwork;
The tactile sensing method, wherein parameters of the first to third subnetworks are updated for each repetition interval so as to output tactile information corresponding to a tactile level mapped to a predetermined user or user group. 15. The method of claim 14,
The first subnetwork is
A tactile sensing method, which is a subnetwork for obtaining a voltage gradient of a time-series touch signal, and includes a one-dimensional convolutional layer and a max pooling layer. 16. The method of claim 15,
The second subnetwork,
A tactile sensing method, which is a sub-network for extracting surface roughness information of a sensing target based on an input frequency signal, and is composed of a deep neural network (DNN). 17. The method of claim 16,
The third subnetwork,
a sub-network integrating the first sub-network and the second sub-network;
A tactile sensing method, which is a sub-network for more precisely classifying tactile information corresponding to a tactile level mapped to a predetermined user or user group. In a computer-readable recording medium recording a program for execution on a computer,
When the program is executed by a processor, the processor
inputting a time-series touch signal and a sliding signal obtained from a detection target into a pre-trained tactile simulation model; and
A computer-readable recording medium comprising executable instructions for outputting tactile information corresponding to a tactile level mapped to a predetermined user or user group.

Images