Towards Haptic Texture Content Library: Texture Synthesis Through Automatic Model Assignment And Texture Authoring in Haptic Attribute Space

PRESENTER: WASEEM HASSAN (2016315589)

ADVISOR: PROF. SEOKHEE JEON, PHD.

Scope of Presentation

Haptic Perception

Kinesthetic

Tactile

Force / torque

Weight

Stiffness

Thermal

Air flow

Through Joints and tendons

Through Skin

Texture

Scenario

Haptics Content Designer

Design haptic feedback (content) for shirts in VR

Friction

Stiffness

Texture

Roughness

Slipperiness

Hardness

Design haptic feedback (content) for shirts in VR

Haptics Content Designer

Physical Properties

Perceptual Properties

Scenario

Design haptic feedback (content) for shirts in VR

Haptics Content Designer

Physics based (parametric equations) method

Data-Driven method

Literature

Physics based (parametric equations) method

Generation of Haptic Contents

Physics based (parametric equations) method [1,2,3,4,5,6]

Every parameter has to be tuned (manual or auto)

Relatively difficult to make realistic

Repeated effort for every new realistic texture

No standards to compare haptic texture models

Advantages

Design control by parameters

Modification is easy

Limitations

Literature

Lateral frictional forces

to render texture

Osgouei, et al. (2019)

Culbertson, et al. (2014)

Record acceleration, force,

and position to render Texture

Jiao, et al. (2018)

Friction and normal force

to render Texture

Ilkhani, et al. (2017)

Record with accelerometer

and play texture

Data-Driven method

Generation of Haptic Contents

Data-Driven method [7,8,9,10]

Need special hardware to collect data

Physical surface is required

Very difficult to make new texture

Modification is not possible

No standards to compare haptic texture models

Advantages

Limitations

Highly realistic

Computationally simple

What are the main problems?

• Lack of Haptic Contents

​

Data-Driven method

Physics based (parametric equations) method

Haptic

Texture Authoring

Automatic

Model Assignment

Haptic

Attribute Space

No standards to compare haptic models

Need special hardware to collect data

Physical surface is required

No standards to compare haptic models

Parameters have to be tuned (manual or auto)

Meaningful Modification is not possible

Repeated for every new texture

Relatively difficult to make realistic

Repeated effort for new realistic texture

Repeated for every new texture

Repeated effort for new realistic texture

Relatively difficult to make realistic

Meaningful Modification is not possible

No standards to compare haptic models

No standards to compare haptic models

What did we do to solve this?

Assign haptic texture based on image [1,2]

Interpolate real textures [4,5]

• Predict haptic attributes from image [3]

• Universal attribute space [3]

1. Hassan, W., Abdulali, A., Abdullah, M., Ahn, S. C., & Jeon, S. (2017). Towards universal haptic library: Library-based haptic texture assignment using image texture and perceptual space. IEEE Transactions on haptics, 11(2), 291-303.
2. Hassan, W., Abdulali, A., & Jeon, S. (2017, June). Perceptual thresholds for haptic texture discrimination. In 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI) (pp. 293-298). IEEE. [Outstanding Paper Award]
3. Hassan, W., Joolee, J.B., & Jeon, S. Towards universal haptic attribute space: Predicting haptic attributes of texture from image features. IEEE Transactions on haptics, [Submission Ready]
4. Hassan, W., Abdulali, A., & Jeon, S. (2019). Authoring new haptic textures based on interpolation of real textures in affective space. IEEE Transactions on Industrial Electronics, 67(1), 667-
5. Hassan, W., Abdulali, A., & Jeon, S. (2018, November). Haptic Texture Authoring: A Demonstration. In International AsiaHaptics conference (pp. 18-20). Springer, Singapore.

Open challenges in Haptics Technology

Haptic Texture Content Library

Automatic Model Assignment

Haptic Attribute Space

Texture Authoring

Using Our System

Haptics Content Designer

Design haptic feedback (content) for shirts in VR

Using Our System

Online

Haptics Content Designer

How does it feel?

Automatic Model

Assignment

Using Our System

Online

Haptics Content Designer

How different are they?

Which one is softer?

Haptic Attribute

Space

Using Our System

Haptics Content Designer

Design Haptic Feedback for shirts in VR

Real Shirt 1

Real Shirt 2

Shirt 1 + Shirt 2

These look good!

Haptic Texture

Authoring

Automatic Haptic Texture Model Assignment

Overall Automation Process

1. A. Abdulali and S. Jeon, “Data-driven modeling of anisotropic haptic textures: Data segmentation and interpolation,” in Proc. Int. Conf. Human Haptic Sens. Touch Enabled Computing. Appl., 2016, pp. 228–239.
2. A. Abdulali and S. Jeon, “Data-driven rendering of anisotropic haptic textures,” in Proc. Int. AsiaHaptics Conf., 2016, pp. 401–407.

Automatic Model Assignment

• Model - a representation of a texture from which its haptic sensation can be recreated

​

Relationship

Limited Haptic models

Infinite Surfaces

Automatic Model Assignment

Image Feature Space

Training

MC-SVM

(RBF Kernel)

Multi-dimension

Perceptual Space

K-means Grouping

Universal Haptio-Visual Texture Library

Image Feature Extraction

Image Feature Selection

Perceptual Space

Image Based Sub classification

New Image

BSIF Features

Unique haptic model Assigned

Training

Testing

Image Feature Extraction

Classification into Perceptually Similar Group

Automatic Model Assignment

84 Texture

Surfaces

Perceptual Space

Psychophysical Experiment

Similarity Rating

3 Dimensional Perceptual Space

K-means clustering with k=16

Automatic Model Assignment

Image Feature Space

84 Texture

Surfaces

GLCM

NGTDM

GLRLM

GLSZM

Gradient

Spatial frequency

(Total 98 features)

Image Feature

Extraction

Linear Regression

P<0.05

Or

features < 10

true

false

Reduced Image

Feature Set

Perceptual Space (3D)

Correlation

Correlation

Comparison

Sequential Forward

Selection

Parallel Analysis

Random Data

Perceptual Space (3D)

Best Ten Features

Automatic Model Assignment

Multi Class Support Vector Machine

Multi-Class Support Vector Machine

MDS K-means (K=16)

Best Ten Features

Universal Haptio-Visual Texture Library

Training MC-SVM

Library

MC-SVM

Image Based Sub classification

New Image

BSIF Features

Unique haptic model Assigned

Image Feature Extraction

Classification into Perceptually Similar Group

Testing MC-SVM

Automatic Model Assignment

Evaluation

21 new texture

Surfaces

Surfaces highlighted in red are the 21 new textures

Automatic Model Assignment

Evaluation

Perceptual Threshold For correct assignment

Average Variance of the Groups

(188.7)

Comparison with other algorithms

Automatic Model Assignment

Demo Presented in:

EuroHaptics 2016 (U.K),

AsiaHaptics 2016 (Japan),

KHC 2017 (South Korea),

EuroHaptics 2018 (Germany),

SIGGRAPH 2019 (U.S)

Modeling [9], Rendering [15]

Haptic Attribute Space

Haptic Attribute Space

For vision we have the RGB

For haptics we have these major dimensions

[11,12,13,14]

Violet

125-0-255

Carpet

Haptic Attribute Space

Roughness

Friction

Hardness

Haptic Attribute Space

Does it feel the same as my shirt?

It is softer than mine?

Haptic Attribute Space

Roughness

Friction

Hardness

Jeans

sandpaper

aluminum

sponge

rubber

acrylic

paper

towel

mesh

wood

Haptic Attribute Space

RESNET50

LBP

GLCM

Image Feature Extraction

Rough - Smooth

Flat - Bumpy

Sticky - Slippery

Hard - Soft

Haptic Attributes Space

1D CNN

Model Training

Image

Physical

Sample

100 Texture

Surfaces

Image Features

1D CNN

Trained Model

Prediction

Rough - Smooth

Flat - Bumpy

Sticky - Slippery

Hard - Soft

Training

Testing

New Image

Haptic Attribute Space

Haptic Attributes Space

100 Texture

Surfaces

Psychophysical experiment

Abrasive Granular Bald bouncy Flat Glassy Hard Cold Grating Warm Pointy Fizzy Sticky Sharp Wavy Wooden Hatched Smooth Jarred Patterned Solid Mild Silky Malleable Prickly Metallic Refined Angular Rigid Rough Jagged Irritating

Slippery Mushy Slick Furry Grainy Pleasant Bumpy Spongy Bubbly Thick Fine Soft

​

​

List of Attributes

Attribute values for the selected attribute pairs

Haptic Attribute Space

Haptic Attributes Space

4D Haptic attribute space

Haptic Attribute Space

Image Feature Space

100 Texture

Surfaces

ResNet50

LBP

GLCM

Haptic Attribute Space

Multi-Scale 1D-CNN

Loss Function: MSE

Optimizer: ADAM

Activation Function: Sigmoid

Haptic Attribute Space

Evaluation

Leave-one-out Cross Validation

Haptic Attribute Space

Evaluation

Mean Absolute Error (MAE)

Linear Regression

Support Vector Regression

State of the Art

1D-CNN [16]

Haptic Attribute Space

Evaluation

Root Mean Square Error (RMSE)

Haptic Texture Authoring

Haptic Texture Authoring

4D Haptic Attribute Space

Haptic Texture Authoring

R-S = 20

F-B = -13

S-S = 5

Carpet

Plastic mesh

R-S = -31

F-B = 41

S-S = -24

Glitter paper

R-S = -6

F-B = -35

S-S = 16

H-S = 37

H-S = -19

H-S = -3

R-S = 20

F-B = 41

S-S = 16

H-S = -3

Haptic Texture Authoring

To modify haptic textures at will

Haptic Texture Authoring

Haptic Model Space

Haptic Texture Models

Acceleration Patterns

Authoring Space

Mel Frequency Cepstral

Coefficients (MFCC) Features

Feature Selection, Reduction, and

Transformation

Multi-dimensional

Perceptual Space

Attribute Rating

Affective Properties

(Hardness Roughness)

Affective Space

Rendering using Weighted Synthesization

Haptic Rendering

Interpolation

in Authoring Space

Haptic Texture Authoring

Affective Space

2D Perceptual Space

Similarity Rating

Psychophysical Experiment

Similarity Rating

Experiment 1

Haptic Texture Authoring

Affective Space

Attribute Rating

Psychophysical Experiment

Attribute Rating

Experiment 2

Haptic Texture Authoring

Top two Attributes Regressed into Perceptual Space

Affective Space

Correlation between attributes and perceptual space

Attribute Rating

Perceptual Space

Correlation

Haptic Texture Authoring

Affective Space

The perceptual space projected onto the attribute lines

Haptic Texture Authoring

25 Acceleration signals (10 secs) from each texture

(Five scan velocities and five contact force)

25 Real-life textured surfaces

Haptic Model Space

Haptic Texture Authoring

Authoring Space

Acceleration signals

Mel Frequency Cepstral

Coefficients (MFCC) Features

​

25 surfaces x (25 signals x 13 coefficients)

Linear Regression

P<0.05

Or

features < 10

true

false

Affective Space (2D)

Correlation

Correlation

Comparison

Sequential Forward

Selection

Parallel Analysis

Random Data

Affective Space (2D)

Best Features

Reduced Image

Feature Set

PCA

One feature for Hard-Soft

One feature for Rough-Smooth

Haptic Texture Authoring

Authoring Space

-0.15

-0.95

-0.5

0.6

Haptic Texture Authoring

Interpolation in Authoring Space and Rendering

Inverse distance Interpolation of 3 textures (S2, S24, S8) to render S25

Haptic Texture Authoring

Evaluation

• 6 textures removed from the space
• Interpolated from three nearest neighbors
• The original and interpolated are compared by humans

Haptic Texture Authoring

Evaluation

Low score means a better match

Realism score normalized according to the reference comparisons

Haptic Texture Authoring

Demo Presented in:

AsiaHaptics 2018 (Korea),

Haptics Symposium 2018 (U.S),

SIGGRAPH 2019 (U.S)

Modeling [9], Rendering [15]

Conclusions

Haptic Texture Model from image

Extract haptic attributes from image

+

Combine real textures to make a virtual one

Universal Haptic Attribute Space

to standardize haptic information

Future Directions

• To combine all the three systems into a unified model
• Assigning model, authoring, and haptic attributes

​

​

• Extending the automation process to other haptic properties
• Shape, compliance etc.

​

Thank you

1. Hassan, W., Abdulali, A., Abdullah, M., Ahn, S. C., & Jeon, S. (2017). Towards universal haptic library: Library-based haptic texture assignment using image texture and perceptual space. IEEE Transactions on Haptics, 11(2), 291-303. [IF 2.487]
2. Hassan, W., Abdulali, A., & Jeon, S. (2019). Authoring new haptic textures based on interpolation of real textures in affective space. IEEE Transactions on Industrial Electronics, 67(1), 667-676. [IF 8.235]
3. Hassan, W., Kim, H., Talhan, A., & Jeon, S. (2020). A Pneumatically-Actuated Mouse for Delivering Multimodal Haptic Feedback. Applied Sciences, 10(16), 5611. [IF 2.679]
4. Hassan, W., Raza, A., Abdullah, M., Jeon, S., “Apparatus for con-trolling electronic function module in the vehicle using steering wheel with dual ubiquitous haptic sensor (듀얼 유비쿼터스 햅틱 센서가 적용된 스티어링 휠을 이용한 차량 내 전장제어 장치).” South Korean patent 1022757610000, registered July 5, 2021
5. Hassan, W., Raza, A., Abdullah, M., Shadman, H.Md., Jeon, S., HapWheel: Bringing in-Car Controls to Driver's Fingertips by Embedding Ubiquitous Haptic Displays into a Steering Wheel. IEEE Transactions on Intelligent Transportation Systems [Under Revision] [IF 6.492]
6. Hassan, W., Joolee, J.B., & Jeon, S. Towards universal haptic attribute space: Predicting haptic attributes of texture from image features. IEEE Transactions on Haptics, [Submission Ready] [IF 2.487]
7. Hassan, W., Abdulali, A., & Jeon, S. (2017, June). Perceptual thresholds for haptic texture discrimination. In 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI) (pp. 293-298). IEEE. [Outstanding paper award]
8. Hassan, W., Abdulali, A., & Jeon, S. (2018, November). Haptic Texture Authoring: A Demonstration. In International AsiaHaptics conference (pp. 18-20). Springer, Singapore.

References

1. S. Jeon and S. Choi, “Real stiffness augmentation for haptic augmented reality,” Presence: Teleoperators Virtual Environments, vol. 20, no. 4, pp. 337–370, 2011.
2. T. Yamamoto, B. Vagvolgyi, K. Balaji, L. L. Whitcomb, and A. M. Okamura, “Tissue property estimation and graphical display for teleoperated robot-assisted surgery,” in Proc. IEEE Int. Conf. Robot. Autom., 2009, pp. 4239–4245
3. C. H. Ho, C. Basdogan, and M. A. Srinivasan, “Efficient point-based rendering techniques for haptic display of virtual objects,” Presence, vol. 8, no. 5, pp. 477–491, 1999
4. M. A. Costa, M. R. Cutkosky, and S. Lau, “Roughness perception of haptically displayed fractal surfaces,” in Proc. ASME Dyn. Syst. Control Divis., 2000, pp. 1073–1079.
5. J. P. Fritz and K. E. Barner, “Stochastic models for haptic texture,” in Proc. Telemanipulator Telepresence Technologies III, 1996, pp. 34–45.
6. L. Kim, A. Kyrikou, G. S. Sukhatme, andM. Desbrun, “An implicit-based haptic rendering technique,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2002, vol. 3, pp. 2943–2948
7. H. Culbertson, J. Unwin, and K. J. Kuchenbecker, “Modeling and rendering realistic textures from unconstrained tool-surface interactions,” IEEE Trans. Haptics, vol. 7, no. 3, pp. 381–393, Jul./Sep. 2014.
8. S. Andrews and J. Lang, “Haptic texturing based on real-world samples,” in Proc. IEEE Int. Workshop Haptic, Audio Visual Environments Games, 2007, pp. 142–147.
9. A. Abdulali and S. Jeon, “Data-driven modeling of anisotropic haptic textures: Data segmentation and interpolation,” in Proc. Int. Conf. Human Haptic Sens. Touch Enabled Comput. Appl., 2016, pp. 228–239.
10. Y. Ujitoko and Y. Ban, “Vibrotactile signal generation from texture images or attributes using generative adversarial network,” in Proc. Int. Conf. Human Haptic Sens. Touch Enabled Comput. Appl., 2018, pp. 25–36.
11. M. Yoshida, “Dimensions of tactual impressions (1),” Jpn. Psychological Res., vol. 10, no. 3, pp. 123–137, 1968.
12. R. H. LaMotte, “Softness discrimination with a tool,” J. Neurophysiology, vol. 83, no. 4, pp. 1777–1786, 2000.

​

References

13. M. Hollins, F. Lorenz, A. Seeger, and R. Taylor, “Factors contributing to the integration of textural qualities: Evidence from virtual surfaces,” Somatosensory Motor Res., vol. 22, no. 3, pp. 193–206, 2005.
14. F. Shao, X. Chen, C. Barnes, and B. Henson, “A novel tactile sensation measurement system for qualifying touch perception,” Proc. Inst. Mech. Eng., Part H: J. Eng. Med., vol. 224, no. 1, pp. 97–105, 2010
15. A. Abdulali and S. Jeon, “Data-driven rendering of anisotropic haptic textures,” in Proc. Int. AsiaHaptics Conf., 2016, pp. 401–407.
16. ] G. T. Taye, H.-J. Hwang, and K. M. Lim, “Application of a convolutional neural network for predicting the occurrence of ventricular tachyarrhythmia using heart rate variability features,” Scientific reports, vol. 10, no. 1,pp. 1–7, 2020
17. Shin, Sunghwan, and Seungmoon Choi. "Geometry-based haptic texture modeling and rendering using photometric stereo." In 2018 IEEE Haptics Symposium (HAPTICS), pp. 262-269. IEEE, 2018.
18. Halabi, Osama, and Gulrukh Khattak. "Generating haptic texture using solid noise." Displays 69 (2021): 102048.
19. Meyer, David J., Michael A. Peshkin, and J. Edward Colgate. "Tactile paintbrush: A procedural method for generating spatial haptic texture." In 2016 IEEE Haptics Symposium (HAPTICS), pp. 259-264. IEEE, 2016.
20. Osgouei, Reza Haghighi, Jin Ryong Kim, and Seungmoon Choi. "Data-driven texture modeling and rendering on electrovibration display." IEEE transactions on haptics 13, no. 2 (2019): 298-311.
21. Culbertson, Heather, Juliette Unwin, and Katherine J. Kuchenbecker. "Modeling and rendering realistic textures from unconstrained tool-surface interactions." IEEE transactions on haptics 7, no. 3 (2014): 381-393.
22. Jiao, Jian, Yuru Zhang, Dangxiao Wang, Yon Visell, Dekun Cao, Xingwei Guo, and Xiaoying Sun. "Data-driven rendering of fabric textures on electrostatic tactile displays." In 2018 IEEE Haptics Symposium (HAPTICS), pp. 169-174. IEEE, 2018.
23. Ilkhani, Gholamreza, Mohammad Aziziaghdam, and Evren Samur. "Data-driven texture rendering on an electrostatic tactile display." International Journal of Human–Computer Interaction 33, no. 9 (2017): 756-770.

​

​

Supplementary Slides

61

Evaluation

• 21 new – outside library – texture surfaces
• Automatic assignment of models to new surfaces
• Psychophysical experiment – to check accuracy

​

62

New surfaces used for evaluation

Incorrectly Assigned Samples

S1

S100

Angles for adjective pairs

ANN

FC 200

Regression Output Layer

Input Features

Predicted

Adjective Rating

FC 100

Individual Feature Prediction (1D-CNN)

Individual Feature Prediction (1D-CNN)

Generation of Haptic Contents

photometric stereo

Dahl model for stiffness

Generation of Haptic Contents

1. Degraen, Donald, Michal Piovarči, Bernd Bickel, and Antonio Krüger. "Capturing Tactile Properties of Real Surfaces for Haptic Reproduction." In The 34th Annual ACM Symposium on User Interface Software and Technology, pp. 954-971. 2021.

photometric stereo

Generation of Haptic Contents

• Culbertson, et al. (2014)

Tool with acceleration, force, and position sensor

Generation of Haptic Contents

• Jiao, et al. 2018.

Friction on electrostatic display

Generation of Haptic Contents

• Ilkhani, et al. 2017

Record (accelerometer) and play texture

Electrostatic Tactile Display

Generation of Haptic Contents

1. Halabi et al. 2021

Perlin’s noise equation

What did we do to solve this?

Assign haptic texture based on image [1,2]

Interpolate real textures [4,5]

• Predict haptic attributes from image [3]

• Universal attribute space [3]

1. Hassan, W., Abdulali, A., Abdullah, M., Ahn, S. C., & Jeon, S. (2017). Towards universal haptic library: Library-based haptic texture assignment using image texture and perceptual space. IEEE transactions on haptics, 11(2), 291-303.
2. Hassan, W., Abdulali, A., & Jeon, S. (2017, June). Perceptual thresholds for haptic texture discrimination. In 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI) (pp. 293-298). IEEE.
3. Hassan, W., Joolee, J.B., & Jeon, S. Towards universal haptic attribute space: Predicting haptic attributes of texture from image features. IEEE transactions on haptics, [Submission Ready]
4. Hassan, W., Abdulali, A., & Jeon, S. (2019). Authoring new haptic textures based on interpolation of real textures in affective space. IEEE Transactions on Industrial Electronics, 67(1), 667-
5. Hassan, W., Abdulali, A., & Jeon, S. (2018, November). Haptic Texture Authoring: A Demonstration. In International AsiaHaptics conference (pp. 18-20). Springer, Singapore.

Open challenges in Haptics Technology

Haptic Texture Content Library

Automatic Model Assignment

Haptic Attribute Space

Texture Authoring

Using Our System

Online

Haptics Content Designer

How does it feel?

How different are they?

Using Our System

Haptics Content Designer

Design Haptic Feedback for shirts in VR

Real Shirt 1

Real Shirt 2

Shirt 1 + Shirt 2

These look good!

Generation of Haptic Contents

Physics based (parametric equations) method

Data-Driven method

Relatively difficult to make realistic

Highly realistic

Computationally simple

Can be computationally expensive to make realistic

Parameters provide higher control

Automatic Model Assignment

• Automatic selection of haptic model to a surface based on image features