Journal of Jilin University(Engineering and Technology Edition) ›› 2026, Vol. 56 ›› Issue (10): 2524-2534.doi: 10.13229/j.cnki.jdxbgxb.20250497

Previous Articles    

Pseudo⁃haptic feedback for multimodal in⁃vehicle interaction

Chuan-liang SHEN(),Rui-zhang YE,Xiao-yuan MA,Jia-yao ZHANG,Zhen WANG,Hong-yu HU()   

  1. National Key Laboratory of Automitive Chasis Integration and Bionics，Jilin University，Changchun 130022，China
  • Received:2025-10-21 Online:2026-10-01 Published:2026-10-09
  • Contact: Hong-yu HU E-mail:shencl@jlu.edu.cn;huhongyu@jlu.edu.cn

RICH HTML

 2   

PDF (PC)

7

Abstract:

This study aims to enhance the feedback efficiency and perceptual consistency of in-vehicle interaction systems under complex driving conditions by developing a pseudo-haptic signal system that integrates visual， auditory， and tactile modalities. The system is built using a capacitive touchscreen and piezoelectric actuators， with a designed vibration parameter matrix to model the mapping between perceptual attributes and vibration features. Kriging interpolation is employed to construct three-dimensional perceptual surfaces. Two signal generation methods—frequency mapping and harmonic envelope modulation—are proposed， with the latter showing better performance in frequency alignment and signal discriminability. A recognition experiment was conducted using four levels of stimulus quantity. Results indicate that the visual modality achieved the highest identification performance， while the tactile modality was more sensitive to changes in stimulus number. A 3.2 km driving simulation scenario was constructed to represent typical driving conditions. Under this setup， the auditory-tactile concurrent masking configuration（A+H（+V）） yielded the lowest RMS and peak yaw rates across most conditions， indicating the highest driving stability under this secondary task， along with the best overall performance in terms of response time and subjective evaluation.

Key words: computer application, pseudo-haptics, multimodal feedback, in-vehicle interaction, harmonic envelope modulation, masking technology

CLC Number: 

  • U461

Fig.1

Haptic feedback device"

Fig.2

Synchronized multimodal signal transmission system"

Fig.3

3D surface plot of physical-perceptual attribute relationship"

Fig.4

Auditory-haptic frequency mapping method"

Fig.5

First-50 harmonic envelope superposition （First 0.01 s）"

Fig.6

Auditory-haptic signal pair comparison across three methods"

Fig.7

Least-confused auditory-haptic signal pairs using harmonic envelope method"

Fig.8

Intermodal identification accuracy fitted curve"

Fig.9

Intermodal mean recognition time fitted curve"

Fig.10

Intermodal information transfer（IT） curve and theoretical upper bound"

Table 1

Multimodal masking feedback experimental conditions"

编号实验条件简写（V/A/H）
1视V
2听（+视）A（+V）
3触（+视）H（+V）
4听+触（+视）共发掩蔽A+H（+V）共发掩蔽
5听+触（+视）后向掩蔽A+H（+V）后向掩蔽
6听+触（+视）基座掩蔽A+H（+V）基座掩蔽
7仅驾驶任务无次任务仅驾驶任务

Fig.11

Multimodal masking stimulus feedback temporal comparison diagram"

Fig.12

Driving simulator scenario and multimodal secondary task design"

Fig.13

Four-condition vehicle dynamics RMS and peak analysis"

Fig.14

Group-wise reductions in RMS and peak values relative to visual-only condition（V）"

Fig.15

Driving and secondary task efficiency and workload evaluation"

[1] Gao F, Ge X, Li J, et al. Intelligent cockpits for connected vehicles: taxonomy, architecture, interaction technologies, and future directions[J]. Sensors, 2024, 24(16): 5172.
[2] Kun A L, Schmidt A, Dey A, et al. Automotive user interfaces and interactive applications in the car[J]. Personal and Ubiquitous Computing, 2013, 17(5): 801-802.
[3] Prabhakar G, Biswas P. A brief survey on interactive automotive UI[J]. Transportation Engineering, 2021, 6: 100089.
[4] Erolin J. Haptic Feedback Defined: Types, Benefits & Uses[EB/OL]. [2026-04-13]. .
[5] Ng A, Brewster S A. Investigating pressure input and haptic feedback for In-car touchscreens and touch surfaces[C]∥Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, New York, USA: 2016: 121-128.
[6] 马晓娜, 童倩倩, 李一含, 等. 基于多模态触觉与伪触觉反馈的智能设计触觉体验构建研究[J]. 装饰, 2022(9): 28-33.
Ma Xiao-na, Tong Qian-qian, Li Yi-han, et al. Research on the construction of haptic experience in intelligent design based on multimodal haptics and pseudo-haptic feedback[J]. Art & Design, 2022(9): 28-33.
[7] Lecuyer A, Coquillart S, Kheddar A, et al. Pseudo-haptic feedback: can isometric input devices simulate force feedback?[C]∥Proceedings IEEE Virtual Reality 2000(Cat. No.00CB37048), New Brunswick, USA, 2000: 83-90.
[8] Hachisu T, Cirio G, Marchal M, et al. Pseudo-haptic feedback augmented with visual and tactile vibrations[C]∥IEEE International Symposium on VR Innovation, Singapore, 2011: 327-328.
[9] Magnusson C, Rassmus-Gröhn K. A pilot study on audio induced pseudo-haptics[C]∥Proceedings of the 3rd International Haptic and Auditory Interaction Design Workshop, Jyväskylä, Finland, 2008: 6-7.
[10] De-Pra Y, Catrambone V, van-Wassenhove V, et al. Altering time perception in virtual reality through multimodal visual-tactile kappa effect[J]. IEEE Transactions on Haptics, 2023, 16(4): 518-523.
[11] Jiang P, Kent C, Rossiter J. Towards sensory substitution and augmentation: mapping visual distance to audio and tactile frequency[J]. PLOS One, 2024, 19(3): e0299213.
[12] Zhu A, Choi A T H, Ma K H, et al. Tactile vibrating toolkit and driving simulation platform for driving-related research[J]. Journal of Visualized Experiments: JoVE, 2020(166): e61408.
[13] Miyoshi J K, Chen Q, Li J, et al. Effects of instructional vibrotactile patterns on takeover performance in autonomous driving[C]∥International Conference on Consumer Electronics, Taiwan, China, 2023: 377-378.
[14] Gibson J J. Observations on active touch[J]. Psychological Review, 1962, 69(6): 477-491.
[15] Vega-Bermudez F, Johnson K O, Hsiao S S. Human tactile pattern recognition: Active versus passive touch, velocity effects, and patterns of confusion[J]. Journal of Neurophysiology, 1991, 65(3): 531-546.
[16] Smith A M, Chapman C E, Donati F, et al. Perception of simulated local shapes using active and passive touch[J]. Journal of Neurophysiology, 2009, 102(6): 3519-3529.
[17] Ackerley R, Hassan E, Curran A, et al. An fMRI study on cortical responses during active self-touch and passive touch from others[J]. Frontiers in Behavioral Neuroscience, 2012, 6: 51.
[18] Cybulska-Klosowicz A, Tremblay F, Jiang W, et al. Differential effects of the mode of touch, active and passive, on experience-driven plasticity in the S1 cutaneous digit representation of adult macaque monkeys[J]. Journal of Neurophysiology, 2020, 123(3): 1072-1089.
[19] He B, Bian N, Wu Y, et al. Multimodal interaction strategies for take-over requests in L3 automated driving[C]∥The 9th International Symposium on Sensors, Mechatronics, and Automation System(ISSMAS 2023), Nanjing, China,2023: 421-427.
[20] Ahmaniemi T, Marila J, Lantz V. Design of dynamic vibrotactile textures[J]. IEEE Transactions on Haptics, 2010, 3(4): 245-256.
[21] Strohmeier P, Hornbæk K. Generating haptic textures with a vibrotactile actuator[C]∥Proceedings of the CHI Conference on Human Factors in Computing Systems, Denver, USA, 2017: 4994-5005.
[22] Wu S, Sun X, Wang Q, et al. Tactile modeling and rendering image-textures based on electrovibration[J]. The Visual Computer, 2016, 33(5): 637-646.
[23] Wang Y, Akeju O V, Zhao T. Interpolation of spatially varying but sparsely measured geo-data: a comparative study[J]. Engineering Geology, 2017, 231: 200-217.
[24] Duc M L, Bilik P, Martinek R. Harmonics signal feature extraction techniques: a review[J]. Mathematics, 2023, 11(8): 1877.
[25] Zhang J. Music feature extraction and classification algorithm based on deep learning[J]. Scientific Programming, 2021, 2021(1): 1651560.
[26] Ludeña-Choez J, Gallardo-Antolín A. Feature extraction based on the high-pass filtering of audio signals for acoustic event classification[J]. Computer Speech & Language, 2015, 30(1): 32-42.
[27] Bennasar M, Hicks Y, Setchi R. Feature selection using joint mutual information maximisation[J]. Expert Systems with Applications, 2015, 42(22): 8520-8532.
[1] Xing-wang WANG,Jin-kuan LUO,Zi-yang ZHANG,Yu-pu CHI,Jin-you JIANG,Mei-ming YU. Algorithm of coverage path planning for unmanned aerial vehicle based on improved DQN [J]. Journal of Jilin University(Engineering and Technology Edition), 2026, 56(9): 2444-2455.
[2] Jia-wei LU,Chang-feng YAN. Sensor ontology matching method based on genuine⁃pseudo triplet neural network [J]. Journal of Jilin University(Engineering and Technology Edition), 2026, 56(9): 2456-2466.
[3] Qing-lin AI,Yuan-xiao LIU,Jia-hao YANG. Small target swmantic segmentation method based MFF-STDC network in complex outdoor environments [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(8): 2681-2692.
[4] Yu-fei ZHANG,Li-min WANG,Jian-ping ZHAO,Zhi-yao JIA,Ming-yang LI. Robot inverse kinematics solution based on center selection battle royale optimization algorithm [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(8): 2703-2710.
[5] Yan PIAO,Ji-yuan KANG. RAUGAN：infrared image colorization method based on cycle generative adversarial networks [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(8): 2722-2731.
[6] Xiang-jiu CHE,Liang LI. Graph similarity measurement algorithm combining global and local fine-grained features [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(7): 2365-2371.
[7] Wen-hui LI,Chen YANG. Few-shot remote sensing image classification based on contrastive learning text perception [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(7): 2393-2401.
[8] Shan-na ZHUANG,Jun-shuai WANG,Jing BAI,Jing-jin DU,Zheng-you WANG. Video-based person re-identification based on three-dimensional convolution and self-attention mechanism [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(7): 2409-2417.
[9] Qiong-xin LIU,Tian-tian WANG,Ya-nan WANG. Non-dominated sorted particle swarm genetic algorithm to solve vehicle location routing problems [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(7): 2464-2474.
[10] Jian WANG,Chen-wei JIA. Trajectory prediction model for intelligent connected vehicle [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(6): 1963-1972.
[11] Xiang-jiu CHE,Yu-peng SUN. Graph node classification algorithm based on similarity random walk aggregation [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(6): 2069-2075.
[12] Feng-feng ZHOU,Zhe GUO,Yu-si FAN. Feature representation algorithm for imbalanced classification of multi⁃omics cancer data [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(6): 2089-2096.
[13] Ping-ping LIU,Wen-li SHANG,Xiao-yu XIE,Xiao-kang YANG. Unbalanced image classification algorithm based on fine⁃grained analysis [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(6): 2122-2130.
[14] Hai-peng CHEN,Shi-bo ZHANG,Ying-da LYU. Multi⁃scale context⁃aware and boundary⁃guided image manipulation detection method [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(6): 2114-2121.
[15] Zi-hao SHEN,Yong-sheng GAO,Hui WANG,Pei-qian LIU,Kun LIU. Deep deterministic policy gradient caching method for privacy protection in Internet of Vehicles [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(5): 1638-1647.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright © Journal of Jilin University(Engineering and Technology Edition), All Rights Reserved.
Tel: +86-431-85095297　Fax: +86-431-85094074　E-mail: xbgxb@jlu.edu.cn
Powered by Beijing Magtech Co. Ltd