Gestures can control cranes

Haptic control is proven for cranes, by using a Microsoft Kinect system.


A loader crane operator using a Microsoft Kinect sensor can control loader crane movements, programming motion with gestures, voice, and recognition algorithms. Matlab/Simulink and C++ programming help. Courtesy: West Pomeranian University of Technology,Loader cranes play a significant role in the industry, but despite seemingly simple construction, they are very difficult to control. Advanced wireless controls may give way to the use of operator hand gestures to control the cranes’ motion. Cranes are used all over the world at building sites, on ships, at transportation depots, at ports, and wherever heavy cargo needs to be lifted. They are also used by emergency services and military personnel (for disarming mines, for example).

To control loader cranes, companies employ skilled operators, rigorously trained to increase the efficiency of their work and to minimize the risk of accidents. One of the factors influencing loader crane control is a natural tendency to oscillate loads. Because of oscillations, it is a challenge even for experienced operators to quickly, safely, and accurately move a load without floating it. Too rapid movements of the boom could lead to load swing, and thus destroy the crane, damaging its mechanisms or causing a roll-over of the vehicle on which the crane is mounted. The next challenge for the loader crane operator is to work with hydraulic levers to lift cargo, which can be located out of sight (behind the wall or much higher). In this case, it is practically impossible for an individual operator to precisely move the cargo to a designated place. Doing this requires moving away from the control interface to check the load or ask for help from other people, which in turn reduces safety and productivity.

Crane control

In recent years, despite new technology, construction cranes, control algorithms, and methods for damping load oscillations, little attention has been paid to crane control interfaces and their influence on operator efficiency. The control method of most cranes has not changed for years. In most cases, levers are connected directly to the hydraulic system, control panels, and joysticks. These interfaces receive operator commands, such as pressing a button, which sends a signal to the computer, where it is processed using a suitable algorithm. Then a control signal is sent to the engine, for example, to rotate the crane base.

A recent market trend is the use of radio remote control panels to control loading cranes. Among equipment loader cranes, spider cranes, and forest cranes, remote control panels increasingly are being offered as a standard feature, not an option. Remote control panels are the most advanced control equipment designed for crane handling, equipped with joysticks, electronic levers, buttons, switches, and LED and LCD displays. These elements allow for more precise control, perform more accurate moves, and set many crane parameters without the operator having to enter the cabin or go to the console. Currently, manufacturers are working on increasing the amount of data exchanged between the remote controllers and the crane system, which are shown on the LCD display. In the latest panels, functions of the lever or joystick are shown on the LCD display and change automatically depending on the mode.

Remote control

Control loader cranes with a radio remote control panel are more effective and safer. This method allows one operator to attach a load to the hook and move it to the designated place. This is possible because the operator is free to move with the wireless control panel and can send a command to the crane at any time. While hitching a load, the operator can approach it and then go to a place where the cargo will be moved. In this way, the operator always has a clear view of the load and the environment, minimizing the risk of error or accident. The "Fassi SmartApp" is an Android OS that allows a user to read status information and control the loader crane with a smartphone. As of spring 2012, it was still in the testing phase.

Previously described methods of control by loading cranes are very intuitive and based primarily on the use of levers and joysticks. In addition, they require the operator to learn particular movements and to remember large amounts of information (identification LEDs, switches, shares caused by the deflection of the joystick), and they do not reflect the movement of the load.

Hand gesture control cranes

Loader cranes can be controlled by an interface based on the operator’s gestures and tracking movements. Proof of concept uses the Microsoft Kinect motion controller for tracking the operator. The Microsoft Kinect motion controller receives the spatial positions of 20 points on the body. The data is obtained for control and gesture recognition algorithms.

The system was implemented using the Matlab/Simulink environment, using the "Natural Interaction Device" package, Kinect SDK and Developer Toolkit, and C++ language. Two methods to control the loader crane have been devised. The first uses the Microsoft Kinect sensor to track the position of the operator's hand. The algorithm using inverse kinematics processes the obtained data with hand location. This allows the operator to control the end of the jib crane directly with hand movements. It is a natural and intuitive way to control, because it imitates the daily action of moving objects.

The second control method uses hand tracking and gesture recognition. Gestures signal various crane functions, for example, ejecting mainstays, controlling individual cylinders, or returning the crane to the transport position. Future plans include further system development with new gestures, the location of the crane operator, and the addition of voice commands.

As shown, the growth in computing power, virtual reality, 3D environments, and the use of gestures and human senses to control all types of devices (haptic feedback) can be used to change the way cranes are controlled, or create virtual environments for remote control of loader cranes. The manufacturers of these machines are increasingly recognizing that virtual technologies can move beyond the world of entertainment to play an important role in industry.

- Krzysztof Pietrusewicz and Karol Miądlicki are from West Pomeranian University of Technology, Szczecin, and contributors for Control Engineering Poland. Edited by Mark T. Hoske, content manager, CFE Media, Control Engineering, mhoske(at)


Below see links to other related articles.

No comments
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by...
Each year, a panel of Control Engineering editors and industry expert judges select the System Integrator of the Year Award winners.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
Learn how to create value with re-use; gain productivity with lean automation and connectivity, and optimize panel design and construction.
Go deep: Automation tackles offshore oil challenges; Ethernet advice; Wireless robotics; Product exclusives; Digital edition exclusives
Lost in the gray scale? How to get effective HMIs; Best practices: Integrate old and new wireless systems; Smart software, networks; Service provider certifications
Fixing PID: Part 2: Tweaking controller strategy; Machine safety networks; Salary survey and career advice; Smart I/O architecture; Product exclusives
The Ask Control Engineering blog covers all aspects of automation, including motors, drives, sensors, motion control, machine control, and embedded systems.
Look at the basics of industrial wireless technologies, wireless concepts, wireless standards, and wireless best practices with Daniel E. Capano of Diversified Technical Services Inc.
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
This is a blog from the trenches – written by engineers who are implementing and upgrading control systems every day across every industry.
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Control Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.