Interactive and immersive process-level digital twin for collaborative human–robot construction work

Abstract

Human cognition plays a critical role in construction work, particularly in the context of high-level task planning and in-field improvisation. On the other hand, robots are adept at performing numerical computation and repetitive physical tasks with precise motion control. The unstructured and complex nature of construction environments and the inability to maintain tight tolerances in assembled workpieces pose several unique challenges to the wide application of robots in construction work. Thus, the robotization of field construction processes is best conceived as a collaborative human–robot endeavor that takes advantage of both human and robot intelligence as well as robots’ physical operation capabilities to overcome uncertainties and successfully perform useful construction work onsite. This paper proposes an interactive and immersive process-level digital twin (I2PL-DT) system in virtual reality (VR) that integrates visualization and supervision, task planning and execution, and bidirectional communication to enable collaborative human–robot construction work. In this work paradigm, the human worker is responsible for high-level task planning and work process supervision. The robot undertakes workspace sensing and monitoring, detailed motion planning, and physical execution of the work. A drywall installation case study involving imperfect rough carpentry (wall framing) is presented using a KUKA mobile industrial robotic arm emulator. A human-in-the-loop study involving 20 subjects was conducted for system verification and to collect feedback for future improvements. The experimental results show that users can use the system to specify work sequences, select optimal task plans, and perform robot trajectory guidance after simple training and felt positive about the system functions and user experience. The system demonstrates the potential of transitioning the role of construction workers from physical task performers to robot supervisors. In addition, the system establishes a promising framework for construction workers to remotely collaborate with onsite construction robots.

Ci-Jyun Polar Liang

Assistant Professor (Jan 2024)

My research interests include Human-Robot Collaboration, Computer Vision, Reinforcement Learning, BIM, Digital Twins, and Extended Reality.