Enhancing Pointing Gestures of Non-HMD Users in Asymmetric Collocated Mixed Reality Collaboration

Vo, N., Thai, V., Tang, A., and Le, K. (2026). Enhancing Pointing Gestures of Non-HMD Users in Asymmetric Collocated Mixed Reality Collaboration. In Proceedings of the 2026 International Conference on Advanced Visual Interfaces (AVI '26).

Abstract

A common collocated group setting in mixed-reality (MR) collaboration is a person wearing a MR headset (HMD user) and presenting MR contents to audiences who are not provided with such specialized devices (Non-HMD users). In this setting, while Non-HMD users can view the MR environment shown on a large physical display, it still remains challenging for the HMD user to interpret their pointing gesture when they spatially refer to objects in the MR environment. To address this, we designed and evaluated two pointing techniques---SCREEN and SCREEN+SPACE---that support Non-HMD users in referring to MR content. Screen pointing allows users to refer to MR objects by pointing at their representation on the display, while Screen+Space pointing additionally enables direct spatial references toward the HMD user's environment. In a study involving 36 participants (18 pairs) performing an expert-novice collaboration task, both techniques improved communication fluency and reduced cognitive workload. Notably, Screen+Space pointing was preferred over Screen pointing, where Non-HMD users often pointed relative to the HMD user's physical space rather than the screen. This suggests that people naturally treat MR content as embedded in shared space, and that space pointing enables more natural and effective communication in asymmetric MR collaborations.

Materials

BibTeX

@inproceedings{vo2026pointing,
  abstract = {A common collocated group setting in mixed-reality (MR) collaboration is a person wearing a MR headset (HMD user) and presenting MR contents to audiences who are not provided with such specialized devices (Non-HMD users). In this setting, while Non-HMD users can view the MR environment shown on a large physical display, it still remains challenging for the HMD user to interpret their pointing gesture when they spatially refer to objects in the MR environment. To address this, we designed and evaluated two pointing techniques---SCREEN and SCREEN+SPACE---that support Non-HMD users in referring to MR content. Screen pointing allows users to refer to MR objects by pointing at their representation on the display, while Screen+Space pointing additionally enables direct spatial references toward the HMD user's environment. In a study involving 36 participants (18 pairs) performing an expert-novice collaboration task, both techniques improved communication fluency and reduced cognitive workload. Notably, Screen+Space pointing was preferred over Screen pointing, where Non-HMD users often pointed relative to the HMD user's physical space rather than the screen. This suggests that people naturally treat MR content as embedded in shared space, and that space pointing enables more natural and effective communication in asymmetric MR collaborations.},
  note = {In press},
  type = {conference},
  publisher = {ACM},
  year = {2026},
  booktitle = {Proceedings of the 2026 International Conference on Advanced Visual Interfaces (AVI '26)},
  title = {Enhancing Pointing Gestures of Non-HMD Users in Asymmetric Collocated Mixed Reality Collaboration},
  author = {Vo, Nam-Dang and Thai, Van-Vinh and Tang, Anthony and Le, Khanh-Duy},
}