FOLD IT, HOLD IT

You are standing on a piece of floating ice, surrounded by water. With a forceful stamp of your foot, the ice instantly breaks, and you find yourself plunging downward.

This captivating scene unfolds within the virtual world of a head-mounted device. What truly enhances the sense of reality and fear is the tactile feedback you experience when the surface of the ice breaks, evoking the sensation of falling.

Even though it's a virtual experience, the fear sends your heart rate shooting up.

As well as a head device, there is a mat to stand on, and a curved-origami device that is held in the hand.

The equipment that allows people to touch and feel objects in the virtual environment was recently developed by a research team, led by Professor Jiang Hanqing, from Westlake University in Hangzhou, Zhejiang province.

The handheld origami-enabled variable stiffness module enables users to actively sense the feel of different objects in the virtual world, ranging from soft to hard and from positive to negative.

Zhang Zhuang, a member of Jiang's team and a postdoctoral research associate at Westlake University, together with other team members, had a paper published online in Nature Machine Intelligence on May 29, titled Active Mechanical Haptics with High-fidelity Perception for Immersive Virtual Reality, introducing their recent research breakthrough.

In contrast to the typical reliance on visual and auditory cues, with this novel device, the incorporation of touch amplifies the realism of the virtual scenario, allowing for a more immersive and engaging experience.

According to Zhang, Jiang brought up the idea of developing an active mechanical haptic device applying the principle of origami at the end of 2021.

The hand module is the result, Zhang says.

The handheld device, shaped like a ball, has five positions, each corresponding to a finger. Under each position, there are two plastic sheets that are folded like crosses, which can be twisted by a motor.

Once they are pressed vertically, the feedback from the rebound of the plastic sheets will be different, depending on the angle between the curved panels. This change in tactile sensation is transmitted to the brain, which then makes judgments about whether the object being touched is cotton, wood or a steel ball, Zhang explains.

The foot-mat device works in a similar way but uses multiple steel plates arranged in a matrix pattern. Changing the angles of the folded plates makes the device stiffer or softer.

The handheld device can currently simulate the feeling of touching a soft ball, an elastic ball, a wooden ball, and the sensation of crushing an eggshell and the foot device can give people the feeling of walking on a hard surface, grass or ice, according to Zhang.

"If we continue this research, we can create even more realistic sensations with different textures," he adds.

The biggest creative idea in the team's research is the "paper folding". However, the greatest challenge lies in figuring out the best way to fold the materials — they need to achieve both perfect active touch sensations and fit within limited space.

Zhang explains that the team has conducted extensive calculations and experiments to determine the appropriate materials and angles for folding.

It's surprising that these devices don't need expensive or hard-to-find parts. The plastic sheets in the hand device and the steel plates in the foot mat are easily purchased online and can be replaced with flexible materials, Zhang says.

In the final part of the study, the research team wanted to make sure their findings were accurate. They did this by asking people how they felt and by measuring their muscle activity and heart rate.

The graph shows the highest point where the participants' heart rates increased rapidly when they stepped on the virtual ice and it broke. This sudden collapse made them feel scared, as if they were actually falling, Zhang explains.

Even though the devices are designed with fixed shapes, and the origami-inspired basic structure remains the same, the hardware devices can be made bigger or smaller, which means that once they find the right uses for these devices, they could quickly start customizing them.

According to Zhang, their research not only provides a fresh approach and experience for interactive virtual reality but also, holds promise for broadening the applications of VR technology in various fields, such as entertainment, remote operations, medical diagnosis, rehabilitation robotics and treatment for fear of heights.

Interdisciplinary teamwork

The success of the haptic device research can be attributed to the collaborative efforts of Jiang's lab members. The team is deeply engaged in interdisciplinary research. It comprises more than 30 members with diverse backgrounds in fields such as mechanics, engineering, chemical engineering and robotics.

"The backgrounds of our laboratory members may appear varied, but they all share a common thread of mechanics. Whether it's robotics, interaction, or any other field, the fundamental principles are rooted in mechanics," explains Jiang.

Jiang, 48, a Fellow of the American Society of Mechanical Engineers, joined Westlake University as Chair Professor of Mechanical Engineering in 2021 and established his laboratory.

Jiang's expertise lies in engineering mechanics, primarily focusing on applying theoretical principles of mechanics to solve practical problems.

He stumbled upon the field of paper folding by chance and started exploring its applications in mechanics 10 years ago.

"Paper folding has an intriguing aspect: its high reconfigurability. Different folds in a piece of paper result in different performance characteristics. Our research revolves around designing these folds," he says.

"In the past few years, we found curve folding interesting and discovered that a single curve fold can impart a wide range of specific properties."

After working on "origami research" for 10 years and witnessing the rapid growth of the metaverse, Jiang had the idea to combine the two. He realized that folded paper materials can be both soft and hard, depending on how they are folded. By using folded paper-like structures in robots, they can easily adjust their level of softness or hardness as required.

Jiang mentions that their team's passion lies in further refining the current research achievements and their scientific goals. They are currently in the process of selecting suitable partners for commercialization and transforming their work into practical applications.

"Scientific research requires innovation and should have societal impact, making genuine contributions to society," Jiang says.

In the future, the research team will continue their efforts to enhance the perception of multiple senses and recreate a more complete sense of touch. They are working on using folded paper to create larger-scale experiences and interactions in bigger settings.

Jiang says: "For example, in a game where you need to ride a motorcycle, could we directly 'grow' a touchable motorcycle from the ground?"