Gestures can control cranes

Haptic control is proven for cranes, by using a Microsoft Kinect system.
By Krzysztof Pietrusewicz, Karol Miądlicki February 25, 2014

A loader crane operator using a Microsoft Kinect sensor can control loader crane movements, programming motion with gestures, voice, and recognition algorithms. Matlab/Simulink and C++ programming help. Courtesy: West Pomeranian University of Technology,Loader cranes play a significant role in the industry, but despite seemingly simple construction, they are very difficult to control. Advanced wireless controls may give way to the use of operator hand gestures to control the cranes’ motion. Cranes are used all over the world at building sites, on ships, at transportation depots, at ports, and wherever heavy cargo needs to be lifted. They are also used by emergency services and military personnel (for disarming mines, for example).

To control loader cranes, companies employ skilled operators, rigorously trained to increase the efficiency of their work and to minimize the risk of accidents. One of the factors influencing loader crane control is a natural tendency to oscillate loads. Because of oscillations, it is a challenge even for experienced operators to quickly, safely, and accurately move a load without floating it. Too rapid movements of the boom could lead to load swing, and thus destroy the crane, damaging its mechanisms or causing a roll-over of the vehicle on which the crane is mounted. The next challenge for the loader crane operator is to work with hydraulic levers to lift cargo, which can be located out of sight (behind the wall or much higher). In this case, it is practically impossible for an individual operator to precisely move the cargo to a designated place. Doing this requires moving away from the control interface to check the load or ask for help from other people, which in turn reduces safety and productivity.

Crane control

In recent years, despite new technology, construction cranes, control algorithms, and methods for damping load oscillations, little attention has been paid to crane control interfaces and their influence on operator efficiency. The control method of most cranes has not changed for years. In most cases, levers are connected directly to the hydraulic system, control panels, and joysticks. These interfaces receive operator commands, such as pressing a button, which sends a signal to the computer, where it is processed using a suitable algorithm. Then a control signal is sent to the engine, for example, to rotate the crane base.

A recent market trend is the use of radio remote control panels to control loading cranes. Among equipment loader cranes, spider cranes, and forest cranes, remote control panels increasingly are being offered as a standard feature, not an option. Remote control panels are the most advanced control equipment designed for crane handling, equipped with joysticks, electronic levers, buttons, switches, and LED and LCD displays. These elements allow for more precise control, perform more accurate moves, and set many crane parameters without the operator having to enter the cabin or go to the console. Currently, manufacturers are working on increasing the amount of data exchanged between the remote controllers and the crane system, which are shown on the LCD display. In the latest panels, functions of the lever or joystick are shown on the LCD display and change automatically depending on the mode.

Remote control

Control loader cranes with a radio remote control panel are more effective and safer. This method allows one operator to attach a load to the hook and move it to the designated place. This is possible because the operator is free to move with the wireless control panel and can send a command to the crane at any time. While hitching a load, the operator can approach it and then go to a place where the cargo will be moved. In this way, the operator always has a clear view of the load and the environment, minimizing the risk of error or accident. The "Fassi SmartApp" is an Android OS that allows a user to read status information and control the loader crane with a smartphone. As of spring 2012, it was still in the testing phase.

Previously described methods of control by loading cranes are very intuitive and based primarily on the use of levers and joysticks. In addition, they require the operator to learn particular movements and to remember large amounts of information (identification LEDs, switches, shares caused by the deflection of the joystick), and they do not reflect the movement of the load.

Hand gesture control cranes

Loader cranes can be controlled by an interface based on the operator’s gestures and tracking movements. Proof of concept uses the Microsoft Kinect motion controller for tracking the operator. The Microsoft Kinect motion controller receives the spatial positions of 20 points on the body. The data is obtained for control and gesture recognition algorithms.

The system was implemented using the Matlab/Simulink environment, using the "Natural Interaction Device" package, Kinect SDK and Developer Toolkit, and C++ language. Two methods to control the loader crane have been devised. The first uses the Microsoft Kinect sensor to track the position of the operator’s hand. The algorithm using inverse kinematics processes the obtained data with hand location. This allows the operator to control the end of the jib crane directly with hand movements. It is a natural and intuitive way to control, because it imitates the daily action of moving objects.

The second control method uses hand tracking and gesture recognition. Gestures signal various crane functions, for example, ejecting mainstays, controlling individual cylinders, or returning the crane to the transport position. Future plans include further system development with new gestures, the location of the crane operator, and the addition of voice commands.

As shown, the growth in computing power, virtual reality, 3D environments, and the use of gestures and human senses to control all types of devices (haptic feedback) can be used to change the way cranes are controlled, or create virtual environments for remote control of loader cranes. The manufacturers of these machines are increasingly recognizing that virtual technologies can move beyond the world of entertainment to play an important role in industry.

– Krzysztof Pietrusewicz and Karol Miądlicki are from West Pomeranian University of Technology, Szczecin, and contributors for Control Engineering Poland. Edited by Mark T. Hoske, content manager, CFE Media, Control Engineering,


Below see links to other related articles.