Remote Control System with Haptic Feedback

Miyoshi
Miyoshi

Toyohashi University of Technology (TUT) is involved in a project to build and run operational experiments of a remote control research network linking the university with national colleges of technology throughout Japan. This is part of a larger project to establish infrastructure for collaborating with national colleges of technology in research and education. In 2008, TUT and Hakodate National College of Technology jointly built a remote control system with haptic feedback, employing the SINET3 QoS service. To learn more about the system and the results to date, we talked with two members of the Toyohashi University of Technology Department of Production Systems Engineering, Associate Professor Takanori Miyoshi of the System & Control Laboratory, and Assistant Professor Takashi Imamura of the Cooperative Research Facility Center and Instrumentation Systems Laboratory.
(Interview date: July 3, 2009)

Please tell us first about your research fields.

Miyoshi: The System & Control Laboratory of our University is engaged in a wide range of research in the control engineering field, mainly on robotics and vibration control. My own areas of specialty are power assist systems and remote control. A power assist system enables large, heavy objects to be moved with a small force-moving a 100kg object with one finger, for example.

In the remote control field, our research is on coordinating human movements with the movements of a machine in another location. As an example, a human operator can control the grasping actions of a robot arm by performing identical grasping actions in a remote location. With this system, moreover, the human grasping action is accompanied by haptic feedback, which lets the operator experience the actual sensation of the robot arm when it grasps an object. Trying to carry out this kind of complex control in a natural way is what makes this research challenging but also rewarding.

Imamura
Imamura

Imamura: I am involved in research on measuring human movement, in our Instrumentation Systems Laboratory. One of our projects is measuring the body movements of car drivers to determine their degree of fatigue. We are also carrying out basic research for measuring human balance states and using the results in rehabilitation. I am taking part in this project because network use is effective also from the viewpoint of these sensing activities. In addition to these, we have proposed bipedal walking toys for primary school children that can be made from cheap materials, and are holding workshops for learning how to make them.

How did you come to take part in this technology college collaboration project?

Imamura: Around 80 percent of students in our university come from colleges of technology, and we feel it is very important to carry on research and education exchanges with these colleges. We definitely want students at those colleges to come to our university to acquire more advanced knowledge and skills, and are interested in actively promoting joint research with them. In 2007, this collaboration project began in the university as a whole, in an attempt to put such ideals into practice. For our part, we are inviting Japan’s technology colleges through this project to join in building an experimental infrastructure for remote control and obtain hands-on experience with remote control technology. In the first year five colleges joined our project, with the number growing to nine in the following year.

What are some of the specific things you are doing in this project?

Imamura: One is a remote crane operation experiment, using an overhead crane at the university. Actual object is transported by remote operations at a college, performed on paddle levers while observing the results on a PC display. While an operation is being performed, the levers convey haptic feedback to give the operator a sense of the object vibration and weight. In another remote haptic feedback experiment, paddle levers are located in our university and at a college site. When a lever is pushed at the remote site, the hardness of the remote object can be determined by the operational reaction force.

That sounds interesting.

Miyoshi: We had to overcome certain issues, however. Students taking part in the remote control experiments felt the experience was highly worthwhile, but sometimes the images and sound feeds were interrupted due to network problems. The operator would be receiving feedback from the lever, when all of a sudden the lack of a display image would cause a jarring discrepancy between sight and touch. This not only affects the student’s interest but represents a safety issue.

Imamura: We therefore set about making improvements in two aspects. The first was to develop content that could be used as simply and effectively as possible. To this end, we newly developed a remote drill operation system. Operating a local paddle lever causes a remote drill to drill a hole in plywood. This operation only has a single degree of freedom, as the drill moves forward only. The operator receives three kinds of sensory feedback, the visual information from the video images, the tactile sensation of drilling including the initial resistance and the sensation from penetrating all the way through the board, and the drilling sound.

The second area of improvement was in the network environment. With splendid good timing, the NII happened to be looking for participants in a trial of SINET3 QoS service, so we immediately signed up after talking with the TUT Information and Media Center.

Did you have any issues to deal with in setting up the network?

Miyoshi: The biggest issue, as you might expect, was access line quality. The quality of the SINET3 backbone is more than adequate, but we encountered congestion in the links between SINET and the colleges of technology. For the drill experiment, we had the bandwidth of the link to Hakodate National College of Technology increased from 3Mbps to 5Mbps, and obtained dedicated use of one router port for the network used in the experiment.

Have there been benefits from introducing SINET3 QoS service?

Imamura: The benefits have been huge. Previously the audio would often be interrupted, for example, forcing us to use telephone lines for contact in some cases. After introducing the QoS service, we can have smooth dialogs with the remote site using the network alone. As for the video, it used to be rather jerky but with QoS service the frame rate is now approximately twice as high as before.

Incidentally, after the experiment we analyzed the network traffic on the Hakodate National College of Technology side and found that, while the 5Mbps bandwidth was maxed out both before and after introducing QoS service, only after the service was introduced could we carry on the experiments without any delay. I believe this is largely due to the priority control in QoS service. I should add that for this experiment, we received tremendous support from the SINET3 side, such as for the connection between Hakodate National College of Technology and the Hokkaido University node. We appreciate this greatly.

What kind of activities do you have in mind for the future?

Miyoshi: Currently we are looking into remote control using 3D cameras and 3D projectors for a three-dimensional perspective on the controlled object. It should be possible to make the experiences more realistic by providing depth perspective. We also intend to keep adding more linked colleges. And in the future, we would like to expand such activities to overseas schools as well.

Imamura: Projects like these are extremely effective for having students experience leading-edge research at the university level early on. We would like to continue carrying out various such undertakings in cooperation with the Information and Media Center, and look forward to ongoing support from SINET3 as well.

Thank you.