Journal of Jilin University(Engineering and Technology Edition) ›› 2024, Vol. 54 ›› Issue (11): 3274-3282.doi: 10.13229/j.cnki.jdxbgxb.20230020

Previous Articles    

Gesture input method based on transparent keyboard in augmented reality environment

Yu-xin YE(),Luo-jia XIA,Ming-hui SUN()   

  1. College of Computer Science and Technology，Jilin University，Changchun 130012，China
  • Received:2023-01-07 Online:2024-11-01 Published:2025-04-24
  • Contact: Ming-hui SUN E-mail:yeyx@jlu.edu.cn;smh@jlu.edu.cn

RICH HTML

 0   

PDF (PC)

190

Abstract:

In current augmented reality systems， the most popular touch input method suffers from inefficiency and occlusion. To address these challenges， this paper proposes a novel text entry method that combines gesture keyboard with an imaginary keyboard， where users draw gestures in a transparent area. First， we verified that it is feasible and practical to use an imaginary keyboard in an augmented reality environment. Second， we conducted a user study to evaluate the performance of three entry methods： touch， gesture with visual feedback and gesture without visual feedback （the method of this paper）. The results show that the average speed of the proposed method can reach 16.95 WPM （Words Per Minute）， which is 6.92 WPM higher than touch input.

Key words: computer application technology, human-computer interaction, augmented reality, text entry, gesture keyboard

CLC Number: 

  • TP391.9

Fig.1

Interface layout"

Fig.2

Pilot study （The participant was pointing at the position of letter n）"

Fig.3

Imaginary key position distribution with 95% confidence ellipses （The black boundaries are the touch area borders）"

Fig.4

Gesture point sequence of word “this”"

Fig.5

Interface layouts of two comparative methods"

Fig.6

Entry speed comparisons of three entry methods"

Fig.7

Error rate comparisons of three entry methods"

Fig.8

Interactive experience comparisons of three entry methods"

1 Lu F, Davari S, Lisle L, et al. Glanceable AR: evaluating information access methods for head-worn augmented reality[C]∥2020 IEEE Conference on Virtual Reality and 3D User Interfaces(VR), Atlanta, USA, 2020: 930-939.
2 秦贵和, 黄俊锋, 孙铭会. 基于双手键盘的虚拟现实文本输入[J].吉林大学学报: 工学版, 2022, 52(8): 1881-1888.
Qin Gui-he, Huang Jun-feng, Sun Ming-hui. Text input based on two⁃handed keyboard in virtual environment[J]. Journal of Jilin University (Engineering and Technology Edition), 2022, 52(8): 1881-1888.
3 Hincapié R J D, Guo X, Moghadasian P, et al. Consumed endurance: a metric to quantify arm fatigue of mid-air interactions[C]∥Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Toronto, Canada, 2014: 1063-1072.
4 Lages W S, Bowman D A. Walking with adaptive augmented reality workspaces: design and usage patterns[C]∥Proceedings of the 24th International Conference on Intelligent User Interfaces, Marina del Ray, United States, 2019: 356-366.
5 Lee L H, Lam K Y, Yau Y P, et al. Hibey: hide the keyboard in augmented reality[C]∥2019 IEEE International Conference on Pervasive Computing and Communications, Kyoto, Japan, 2019: 1-10.
6 Pourchon R, Léger P M, Labonté-LeMoyne É, et al. Is augmented reality leading to more risky behaviors? An experiment with pokémon go[C]∥International Conference on HCI in Business, Government, and Organizations, Vancouver, Canada, 2017: 354-361.
7 Xu W, Liang H N, He A, et al. Pointing and selection methods for text entry in augmented reality head mounted displays[C]∥2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Beijing, China, 2019: 279-288.
8 Elsayed N A M, Thomas B H, Marriott K, et al. Situated analytics: demonstrating immersive analytical tools with augmented reality[J]. Journal of Visual Languages & Computing, 2016, 36: 13-23.
9 Liverani A, Amati G, Caligiana G. A CAD-augmented reality integrated environment for assembly sequence check and interactive validation[J]. Concurrent Engineering, 2004, 12(1): 67-77.
10 Grossman T, Chen X A, Fitzmaurice G. Typing on glasses: adapting text entry to smart eyewear[C]∥ Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services, Copenhagen, Denmark, 2015: 144-152.
11 Yu C, Sun K, Zhong M, et al. One-dimensional handwriting: Inputting letters and words on smart glasses[C]∥Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, USA, 2016: 71-82.
12 Sridhar S, Feit A M, Theobalt C, et al. Investigating the dexterity of multi-finger input for mid-air text entry[C]∥Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Republic of Korea, 2015: 3643-3652.
13 Ni T, Bowman D, North C. Airstroke: bringing unistroke text entry to freehand gesture interfaces[C]∥ Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Montréal, Canada, 2011: 2473-2476.
14 Wang C Y, Chu W C, Chiu P T, et al. Palmtype: using palms as keyboards for smart glasses[C]∥Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services, Copenhagen, Denmark, 2015: 153-160.
15 Yu D, Fan K, Zhang H, et al. Pizzatext: text entry for virtual reality systems using dual thumbsticks[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(11): 2927-2935.
16 Li F C Y, Dearman D, Truong K N. Virtual shelves: interactions with orientation aware devices[C]∥Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, Victoria, Canada, 2009: 125-128.
17 Yan Y, Yu C, Ma X, et al. Eyes-free target acquisition in interaction space around the body for virtual reality[C]∥Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, Canada, 2018: 1-13.
18 Gustafson S, Bierwirth D, Baudisch P. Imaginary interfaces: spatial interaction with empty hands and without visual feedback[C]∥Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, New York, USA, 2010: 3-12.
19 Lin S Y, Su C H, Cheng K Y, et al. Pub-point upon body: exploring eyes-free interaction and methods on an arm[C]∥Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, Santa Barbara, USA, 2011: 481-488.
20 Cui W, Zheng J, Lewis B, et al. Hotstrokes: word-gesture shortcuts on a trackpad[C]∥Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, United Kingdom, 2019: 1-13.
21 Cui W, Zhu S, Li Z, et al. Backswipe: back-of-device word-gesture interaction on smartphones[C]∥ Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan, 2021: 1-12.
22 Zhai S, Kristensson P O. The word-gesture keyboard: reimagining keyboard interaction[J]. Communications of the ACM, 2012, 55(9): 91-101.
23 Macleod C, Ide N, Grishman R. The American national corpus: a standardized resource for American English[C]∥2nd International Conference on Language Resources and Evaluation, Athens, Greece, 2000: 1-4.
24 Gustafson S, Holz C, Baudisch P. Imaginary phone: learning imaginary interfaces by transferring spatial memory from a familiar device[C]∥Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, Santa Barbara, USA, 2011: 283-292.
25 Kristensson P O, Zhai S. Shark2: a large vocabulary shorthand writing system for pen-based computers[C]∥Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology, Santa Fe, USA, 2004: 43-52.
26 Mackenzie I S, Soukoreff R W. Phrase sets for evaluating text entry techniques[C]∥Chi'03 Extended Abstracts on Human Factors in Computing Systems, Fort Lauderdale, USA, 2003: 754-755.
27 Hart S G, Staveland L E. Development of NASA-TLX (task load index): results of empirical and theoretical research[J]. Advances in Psychology, 1988, 52: 139-183.
28 Arif A S, Stuerzlinger W. Analysis of text entry performance metrics[C]∥2009 IEEE Toronto International Conference Science and Technology for Humanity (TIC-STH), Toronto, Canada, 2009: 100-105.
[1] Xiang-jiu CHE,Yu-ning WU,Quan-le LIU. A weighted isomorphic graph classification algorithm based on causal feature learning [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(2): 681-686.
[2] Li-ming LIANG,Long-song ZHOU,Jiang YIN,Xiao-qi SHENG. Fusion multi-scale Transformer skin lesion segmentation algorithm [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(4): 1086-1098.
[3] Dondrub LHAKPA,Duoji ZHAXI,Jie ZHU. Tibetan text normalization method [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(12): 3577-3588.
[4] Na CHE,Yi-ming ZHU,Jian ZHAO,Lei SUN,Li-juan SHI,Xian-wei ZENG. Connectionism based audio-visual speech recognition method [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(10): 2984-2993.
[5] Ya-hui ZHAO,Fei-yu LI,Rong-yi CUI,Guo-zhe JIN,Zhen-guo ZHANG,De LI,Xiao-feng JIN. Korean⁃Chinese translation quality estimation based on cross⁃lingual pretraining model [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(8): 2371-2379.
[6] Shan XUE,Ya-liang ZHANG,Qiong-ying LYU,Guo-hua CAO. Anti⁃unmanned aerial vehicle system object detection algorithm under complex background [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(3): 891-901.
[7] Zhen WANG,Xiao-han YANG,Nan-nan WU,Guo-kun LI,Chuang FENG. Ordinal cross entropy Hashing based on generative adversarial network [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(12): 3536-3546.
[8] Feng-feng ZHOU,Zhen-wei YAN. A model for identifying neuropeptides by feature selection based on hybrid features [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(11): 3238-3245.
[9] Gui-he QIN,Man-ying WANG,Ming-hui SUN. Research on a pressure-based interaction paradigm for multimedia terminal [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(11): 3186-3193.
[10] Jun-jie WANG,Yuan-jun NONG,Li-te ZHANG,Pei-chen ZHAI. Visual relationship detection method based on construction scene [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(1): 226-233.
[11] Bing ZHU,Zi-wei LI,Qi LI. Building segmentation method of remote sensing image based on improved SegNet [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(1): 248-254.
[12] Gui-he QIN,Jun-feng HUANG,Ming-hui SUN. Text input based on two⁃handed keyboard in virtual environment [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(8): 1881-1888.
[13] Fu-heng QU,Tian-yu DING,Yang LU,Yong YANG,Ya-ting HU. Fast image codeword search algorithm based on neighborhood similarity [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(8): 1865-1871.
[14] Tian BAI,Ming-wei XU,Si-ming LIU,Ji-an ZHANG,Zhe WANG. Dispute focus identification of pleading text based on deep neural network [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(8): 1872-1880.
[15] Ming LIU,Yu-hang YANG,Song-lin ZOU,Zhi-cheng XIAO,Yong-gang ZHANG. Application of enhanced edge detection image algorithm in multi-book recognition [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(4): 891-896.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright © Journal of Jilin University(Engineering and Technology Edition), All Rights Reserved.
Tel: +86-431-85095297　Fax: +86-431-85094074　E-mail: xbgxb@jlu.edu.cn
Powered by Beijing Magtech Co. Ltd