Noah Kuntz
Motivation
Unmanned helicopters are an increasingly valuable robotic platform owing to their flexibility when maneuvering in restricted urban environments. One advantage of this maneuverability is the ability to land in locations without prepared landing area. This suggests that an unmanned helicopter would be ideally suited as a delivery vehicle for a payload needed, at a moments notice, at a site without a prepared landing pad. Manned helicopters are frequently used to transport cargo that is oversize, overweight, or that which is needed very quickly in a particular spot. However cargo transport by a manned helicopter is a dangerous task that requires a great deal of training and entails risk to both the pilot and ground crews [1]. By using an unmanned vehicle to autonomously pick up the cargo, the risks to pilots and ground crews can be eliminated. One cargo suited for deployment by a UAV is the bomb defusing vehicle known as the BomBot, developed by the West Virginia High Technology Consortium [2]. The unmanned helicopter could be used to deploy a BomBot UGV exactly where it was needed rather than requiring manual human deployment.The ultimate mission of this research is broken down as follows. The UAV will navigate to the drop off point using GPS waypoints. Once near the landing zone a suitable area for cargo drop off is determined by mapping the ground with a LIDAR system and applying a landing zone selection algorithm [3][4][5], or by locating a target zone via lights or other fiducials. After a slow descent over the landing zone, the payload will be deployed and taken control of by a local operator. When the payload needs to be retrieved, the UAV will navigate to the GPS coordinates of the target. There the UAV will search for the cargo, and track it in the case of a UGV cargo. Alternately it could be retrieved at a predetermined and marked point.
There are four tasks that need to be completed for the goal of unmanned cargo transport. The first task is to develop a platform that can stably navigate to a GPS waypoint and also do stable relative position and velocity control. The current platform is the SR-100 UAV helicopter from Rotomotion. The autopilot of the SR-100 uses accelerometers, gyros, a magnetometer and a Novatel GPS to control its position and velocity. The resulting autopilot can hover the helicopter to within a one meter radius horizontally and half a meter radius vertically, as well as navigate to GPS waypoints and perform basic auto-takeoff and landing. Command of the helicopter is performed from a ground station computer using 802.11 wireless networking. The SR100 UAV is capable of carrying approximately 19 lbs of payload.
The three tasks left for discussion are the focus of my research. First, the act of visual servoing to the payload UGV (and possible tracking of the payload while it is in use). This includes the locating of a drop off zone either by ground mapping or locating fiducials. If fiducials are used to mark the area in a similar manner as manned helicopters use [1], this task is highly similar to that of locating or tracking the payload that is to be picked up. Second, mechanisms for carrying and deploying the payload. Third, the visual servoing system for picking up the payload.
In my experiments the visual tracking control was tested with a three degree of freedom gantry system known as the Systems Integrated Sensor Test Rig (SISTR), with an additional 2 degrees of freedom provided by a pan-tilt unit. The 3DOF translational gantry represents the movement of the helicopter trying to servo to the target, while the pan-tilt unit allows more rapid tracking to keep the target in view regardless of the pitching of the helicopter that is necessary for its flight. Results show that the 5DOF mechanism was successfully controlled via vision to track the movement of an RC truck with an LED fiducial. The payload transport and deployment system was constructed and flight tested. A semi-autonomous version of the mission was accomplished to demonstrate feasibility, minus the tracking portion of the mission which has thus far only been performed on the gantry.
The cargo pickup system was also tested on the gantry. Velocities recorded from a flight test were replayed by the gantry to simulate helicopter movement, while at the same time the pan tilt unit of the camera was manipulated to track the target. Once the target was suitably close to the camera, a hook mounted on the gantry via another pan tilt unit was moved to attempt pickup of the target. The results of these tests are presented, as well as an examination of the accuracy of the velocity playback conditions.
Figure 1 - SR-100 Autonomous Helicopter
Experimental Setup
Semi-autonomous UGV Deployment Test
As a proof of concept for the carrying and deployment/retrieval of a UGV by a UAV, a partially autonomous test scenario was carried out. In this scenario the UGV, a 1/10th scale RC truck, was transported inside a carrying bay that was mounted on the belly of the SR-100 unmanned helicopter. This required a custom aluminum landing gear that was constructed in order to accommodate the truck being mounted underneath the helicopter. This setup is shown in Fig 5, with a placeholder UGV. The much wider and longer footprint of this custom landing gear also reduces risk of the helicopter tipping over, and helped the SR-100 survive a rough landing caused by a motor failure.Gantry Control System
In order to control both the 3DOF gantry, the 2DOF camera pan tilt unit, and the 2DOF hook pan tilt unit, a system was constructed that used up to three computers and several methods of communication including serial, Ethernet, USB, SCSI, and wireless PWM. This system also uses up to four distinct pieces of software at the same time, namely a LabVIEW host application, LabVIEW real-time, a custom C++ computer vision application using VideoOCX for video capture, and the MATLAB engine. Figure 2 shows block diagrams of this system in its two different forms.
Figure 2 - GANTRY CONTROL SYSTEM BLOCK DIAGRAMS
Velocity Control and Playback
In order to use the SISTR 3DOF gantry for testing of our visual servoing algorithms, it was first necessary to establish sufficiently accurate velocity control of the gantry. For the purposes of our tests an open loop controller provided sufficient accuracy with minimal control loop overhead. The model for the open loop controller was developed by moving the gantry back and forth at ever-increasing speeds by sending higher and higher values to the motor amplifiers until the limits of the motors were reached. This was correlated to speeds derived from encoder data during the tests. Accurate velocity control of the gantry system was only possible in the x and y axes, the horizontal plane. The vertical/z axis could not be reliably controlled at slow speeds because of high friction and poor motor power relative to the weight moved. This could likely be overcome using either a friction model or an improvement in the mechanical construction, however for the purposes of these tests vertical velocity playback can be omitted.For the playback of helicopter velocities during tests of visual cargo pickup it can be assumed that the target vehicle is located on relatively flat ground and that the helicopter is able to maintain its altitude within approximately 0.32 meters accuracy. The height of the loop on the target, shown in Fig 1, is 0.32 m. For the flight data that is replayed on the gantry, the range of vertical motion is 1.84 meters. However the variance of the recorded vertical motion is 0.14 meters. So the assumption of a range of motion of less than 0.32 meters is reasonable except for outlier cases, brief deviations most likely caused by wind gusts. Therefore we replicated the helicopter velocities in only the x and y axes. A 72 second length of hovering data was replayed by outputting the given velocity at each moment to the open loop velocity controller for the gantry. The data was recorded at 24.19 points per second and using precision timers we were able to replay it at 24.71 points per second. This simulated the motion of the helicopter to a degree useful for preliminary validation of the cargo pickup procedure.
Visual Tracking and Control
The core of the visual tracking algorithm is image-based pose regulation. The pixel error between the desired position of the target and its current position is fed through a Jacobian matrix (1) that maps pixel space to Cartesian space. The goal is to reduce that error in order to keep the target centered in the camera's view.
In equation (1) u and v represent the horizontal and vertical pixel coordinates of the target, u and v are the error between the current and the desired coordinates, f is the focal length of the camera in pixels, and z is the distance to the target in centimeters. The Tx,Ty,Tz values are the translational offsets of the gantry, and the wx,wy,wz values are the rotational offsets. Once u and v are calculated from the image, the T value and w values can be found by taking the pseudo inverse of LT and performing matrix multiplication with s . This sort of basic visual servoing is well established in the literature [13]. To move the gantry a PID control loop was used with the open loop velocity control model of the gantry that is discussed later.
The choice of a fiducial to be visually tracked and the method of fiducial extraction were controlled by two criteria: the speed at which the fiducials could be located and the ability to locate them under a variety of lighting conditions. In early tests the lighting condition constraint was ignored and standard visible LEDs were used as fiducials. The input image was thresholded and the centroids of the white regions were found. This simple method of fiducial identification allowed for ”real-time” tracking at near the speed of the video stream from the camera. Though more state of the art methods for object tracking exist, for this particular application there is no reason the target cannot be labeled with lights. The primary design concern is low computation overhead to minimize lag, which this simple method allows.
Figure 2 - Effect of IR Filter on Acquired Images
Lastly, four fiducial LEDs were used for tracking the UGV, because of space constraints only two were used for tracking the loop during cargo pickup operations.
In order to satisfy the criteria of functioning under various light conditions, the plan was to change the fiducial light’s wavelength to infrared and filter out other light. Initially, infrared LEDs were tested, but all the infrared LEDs tested had either limited viewing angle or limited brightness. So the fiducials were changed from LEDs to krypton light bulbs, which emit a significant amount of light in the infrared band as well as visible white light. At the same time, an infrared band-pass filter was placed over the lens of the camera used for the vision processing. Because of the relatively poor reflectance of infrared light by most non-lustrous surfaces, even under bright lighting conditions the krypton bulbs emit far more infrared than most surfaces reflect. Figure 2 shows the effect of the filter on the acquired images and their histograms. A threshold of 170 out of 255 was used in our tests, without the filter there is a large amount of pixels over 170 including many that are not fiducials. The addition of the filter shifts all the pixel intensities well below the threshold, except for those indicating the fiducials.
For the purposes of our tests the vision system needed to work with two backgrounds: a dark-gray asphalt parking lot, and a tan simulated-desert flooring. These backgrounds are the flying field and gantry floor, and are lit by sunlight and bright theater floodlights respectively. In both cases the light source has a significant infrared component. Preliminary video of the fiducials outdoors suggests that thresholding will be able to identify them against the parking lot surface. Extensive tests in the gantry demonstrated that tracking against the pseudo-desert flooring functioned even under the full brightness of infrared-rich theater floodlights.
Figure 3 - Target Loop With Fiducial Lights
Velocity Control and Playback
In order to use the SISTR 3DOF gantry for testing of our visual servoing algorithms, it was first necessary to establish suf- ficiently accurate velocity control of the gantry. For the purposes of our tests an open loop controller provided sufficient accuracy with minimal control loop overhead. The model for the open loop controller was developed by moving the gantry back and forth at ever-increasing speeds by sending higher and higher values to the motor amplifiers until the limits of the motors were reached. This was correlated to speeds derived from encoder data during the tests. Accurate velocity control of the gantry system was only possible in the x and y axes, the horizontal plane. The vertical/z axis could not be reliably controlled at slow speeds because of high friction and poor motor power relative to the weight moved. This could likely be overcome using either a friction model or an improvement in the mechanical construction, however for the purposes of these tests vertical velocity playback can be omitted.For the playback of helicopter velocities during visual cargo pickup testing we can assume that the target vehicle is located on relatively flat ground and that the helicopter is able to maintain its altitude within approximately 0.32 meters accuracy. The height of the loop on the target, shown in Fig 3, is 0.32 m. For the flight data that is replayed on the gantry, the range of vertical motion is 1.84 meters. However the variance of the recorded vertical motion is 0.14 meters. So the assumption of a range of motion of less than 0.32 meters is reasonable except for outlier cases, brief deviations most likely caused by wind gusts. Therefore we replicated the helicopter velocities in only the x and y axes. A 72 second length of hovering data was replayed by outputting the given velocity at each moment to the open loop velocity controller for the gantry. The data was recorded at 24.188 points per second and using precision timers we were able to replay it at 24.708 points per second. This simulated the motion of the helicopter to a degree useful for preliminary validation of the cargo pickup procedure.
Figure 4 - Output of Image Processing
Visual Cargo Pickup
The visual cargo pickup algorithm is an extension of the visual servoing algorithm. The same thresholding and centroids locating operations are performed on the image as is done in the velocity tracking. The effect of these operations is illustrated in Fig 4, where the IR filtered image and the fully processed image are compared. Crosshair markers are placed on the fiducials to visually indicate successful tracking. Instead of controlling the 3DOF gantry to reach the target, velocity from an actual heli- copter flight is replayed on the x and y axes of the gantry. At the same time the program waits for the target to appear in the cameras field of view. When the fiducials appear the pan tilt unit the camera is mounted on is servoed to center the fiducials in its view.Besides servoing the pan tilt unit to center the fiducials, at each iteration the program calculates the distance to the target. This calculation is based on knowledge of the distance between the fiducials and the focal length of the camera. The algorithm considers that distance and how well centered the fiducials are in the camera’s field of view. Then if the target is within the range of the pickup arm and the target is suitably close to centered in the camera’s view, the pan angle of the camera’s pan tilt unit is matched by the hook’s pan tilt unit and the hook is swept forward towards the target. That the fiducials be near the center of the camera’s view is important to ensure that moving the hook arm to the same angle as the pan tilt unit will in fact line up the arm with the target. The pan tilt unit of the hook’s arm is located directly behind the camera’s pan tilt unit so no additional calculations are needed to correctly match the angle of the camera. If the hook makes it through the loop of the target, and the target is lifted off the ground slightly, then it is considered to be successfully picked up.
Figure 5 - Gantry Control System Block Diagrams
Gantry Control System
In order to control both the 3DOF gantry, the 2DOF camera pan tilt unit, and the 2DOF hook pan tilt unit, a system was constructed that used up to three computers and several methods of communication including serial, Ethernet, USB, SCSI, and wireless PWM. This system also uses up to four distinct pieces of software at the same time, namely a LabVIEW host application, LabVIEW real-time, a custom C++ computer vision application using VideoOCX for video capture, and the MATLAB engine. Figure 5 shows block diagrams of this system in its two different forms.For the initial visual tracking tests, two computers were used. A host computer ran the C++ vision processing program and the velocity control loop. This interpreted data from a CCD camera, then calculated the velocities needed to track the target by performing calculations with the image Jacobian by using the MATLAB engine for matrix operations. It then converted these velocities into the 16 bit value for the motor amplifiers by using the open loop velocity controller, and called a LabVIEW application compiled as a DLL to pass these values to a LabVIEW application via a datasocket, which then passed those to the Lab- VIEW real-time computer using Ethernet. The LabVIEW realtime computer controls the gantry’s movements. At the same time the C++ vision processing program is also controlling the pan tilt unit on the gantry’s end effecter by communicating with it over an RS-232 connection.
For the final visual cargo pickup tests, three computers were used. The computer that previously ran everything except for LabVIEW real-time now only uses C++ to play back the recorded helicopter velocities and communicate them to the local LabVIEW which then sends them to the LabVIEW real-time computer. The third computer handles the vision processing and control loops. It also controls the cargo hook which is mounted on the end of an arm attached to an RC servo based pan-tilt unit. This pan tilt unit is mounted behind and slightly below the camera pan tilt unit. This is communicated with via a PC to RC USB interface. Figure 6 shows what this setup physical looks like. It was necessary to use a third computer because the helicopter velocities could not be played back at an accurate rate on the same computer where the vision was running without both processes being slowed down.
Figure 6 - Cargo Pickup Prototype
EXPERIMENTAL RESULTS
Tracking of a Moving Target
Fiducials were mounted on a 1/10 scale RC truck and tracking was performed using the gantry and vision system. A variety of setups were tested: with and without the pan tilt unit, with four fiducials and with only one, with a change in elevation and with level ground, with smooth slow motion of the target and with harsh accelerations. The results of these tests were mainly binary, whether it could follow the target or not. There was also a qualitative element of how well it followed the target, but no repeated quantitative tests were conducted.The overall result was that the system was able to track the truck under each set of setup conditions, after some tuning of the PID gains. The basic criteria for successful tracking was that the target LEDs never leave the view of the camera. If the target vehicle was moved more quickly than the camera could follow, it would fail to track it. Ultimately following a moving target is not essential to autonomous cargo pickup so this task was mainly used to establish the functionality of the visual servoing and gantry control systems, before the more critical problem of picking up the target was tackled.
Semi-autonomous UGV Deployment Test
As a proof of concept for the carrying and deployment/ retrieval of a UGV by a UAV, a partially autonomous test scenario was carried out. In this scenario the UGV, a 1/10th scale RC truck, was transported inside a carrying bay that was mounted on the belly of the SR-100 unmanned helicopter. This required a custom aluminum landing gear that was constructed in order to accommodate the truck being mounted underneath the helicopter. This setup is shown in Fig 7, with a placeholder UGV. The much wider and longer footprint of this custom landing gear also reduces risk of the helicopter tipping over, and helped the SR-100 survive a rough landing caused by a motor failure.The procedure of the proof of concept test scenario is as follows. The helicopter performed autonomous takeoff, was directed to a GPS waypoint, then was directed to autonomously land. The actuated gate of the carrying bay was remotely lowered, and the UGV navigated out of the helicopter by remote control, driven around and then driven back into the bay. At that point the gate was remotely closed, and once again the helicopter performed an autonomous takeoff, was directed back to its starting point, and then directed to autonomously land. The entire test took less than 10 minutes.
The purpose of this test was to show that the UAV helicopter could safely transport the UGV, and if an autonomous UGV was used this test could have easily been completed fully autonomously. However the ultimate goal of this research is to be able to use an unmanned helicopter to autonomously transport any cargo, so the ability of the cargo to navigate into some sort of bay cannot be relied upon. Also it would be preferable not to have to completely land the UAV, for purposes of both speed and flexibility in terms of the pickup site. Therefore a protocol for autonomous cargo pickup was designed and tested. Procedures for the helicopter-based transport of cargo by manned aircraft were examined for background information. The use of light patterns to point to the cargo to be picked up, as well as the hook design and specifications for acceptable pickup zones, all are being taken into consideration for integration as the design of our cargo pickup system progresses [1].
Figure 7 - SR-100 With Cargo Bay and UGV
Figure 8 - Velocity Playback Test Results
Velocity Playback
As part of the testing of autonomous cargo pickup, the 3DOF gantry system was used to partially mimic the behavior of a hovering unmanned helicopter. To show that the velocities were correctly being replayed by the gantry, the velocity of the gantry was continuously measured by taking a derivative of the encoder readings. These are graphed with the input velocities in Fig 8. The measured gantry velocity is noisier than the input because it is the unfiltered derivative of the encoder values. Besides that high frequency noise the fidelity of the velocity playback is high. Video of the motion of the gantry during velocity playback was also qualitatively compared to video of the helicopter during the recorded flight. There was a noticeable correlation between the motions of the simulation and the actual flight.Visual Cargo Pickup
The goal of the visual cargo pickup test was to gauge the reliability of the system under conditions as close as possible to those that would be experienced while mounted on the actual helicopter. To that end the tests were conducted with the helicopter velocities being played back by the gantry to simulate the sort of motion the system tracking system would have to deal with on the actual aircraft. At the same time the gantry floodlights were at their full brightness, so there was the greatest possible risk of false fiducial detection and so that the environment most closely resembled the outdoors on a sunny. Since the real system mounted on the helicopter would only know the cargo’s location to within the GPS’s accuracy of 20 cm, the target was placed in one central position and eight equally distributed positions 20 cm away. Position 9 was considered to be the actual location of the target while the other 8 positions were the possible 20 cm offsets that could occur if GPS misestimated the position in any direction. Figure 10 shows the target locations relative to the portion of helicopter data that was being replayed.Visual cargo pickup was attempted twice for each possible position of the target, for a total of 18 tests, the results are summarized by Figure 9. In 11 of the tests the target was successfully hooked by the computer. During the 4 near-miss tests the hook was swung within centimeters of target’s loop, contacting the outside of the loop but failing to pick up the target. The last 3 attempts either failed to swing at the target or missed completely.
The near-miss events occurred when the pickup system attempted to hook the target while the gantry was replaying a relatively high velocity. Due to 5:1 gear reduction on the servo pan tilt unit, it takes 1.86 seconds to move the 149 degrees of the pickup swing. The cargo pickup program determines that the target is in range and begins the swing, but during those 1.86 seconds the gantry can move out of reach of the target. This pan tilt unit is designed for power at the expense of speed, in future work a faster pan tilt unit will likely be used. Processing lag is believed to have a minimal amount of detrimental effect compared with this extremely long servo lag.
CONCLUSION AND FUTURE WORK
This research has shown the feasibility of using computer vision for the task of autonomous cargo pickup by an unmanned helicopter. The tests presented here are the first steps toward a completely autonomous helicopter-based air cargo transport system. Through the use of hardware in the loop testing and simulation with a 3DOF gantry, we were able to create a realistic test of the vision system without putting an aircraft at risk. Though the success rate of the cargo pickup tests was only 61%, a success rate of 83% or more should be possible through minor changes to the system hardware. The visual cargo pickup system will continue to be refined using the gantry as a test bench until the visual cargo pickup system can be safely deployed on a helicopter.Plans for future work include:
1. Refinement of cargo pickup operations under simulation to increase the success rate. Improvements include use of a faster pan tilt unit, and autonomous detection of whether a pickup attempt failed or succeeded so that the system may make multiple attempts without being reset.
2. Navigation to the cargo’s position based on ground indicator lights, similar to an approach used by manned helicopters [1]. A sequence of lights leads to the cargo while three lines of light fan out after the cargo. Indicator lights do not require an actual prepared helipad, but are designed to be quickly set up at an improvised landing area.
3. Flight testing of the tracking and cargo pickup system on the SR-100 unmanned helicopter.
4. Development and/or implementation of stabilization algorithms for autonomously carrying the payload suspended below the helicopter.
5. Integration of cargo transport with safe landing zone detection, to allow deployment/pickup of the cargo in completely unprepared environments.
These future plans combined with current work will add up to a robust autonomous cargo transport system. Collision avoidance is also a necessary aspect of such a system in order to bridge the gap between pickup and drop off of cargo. That is a task of substantial complexity in and of itself and is not likely to be addressed by this project. However developing a capability for an unmanned helicopter beyond just surveillance missions enhances the ultimate usefulness of other work related to UAV navigation and control.
Figure 10 - Graph of Gantry Position and Target Positions
REFERENCES
1. Multiservice Helicopter Sling Load: Basic Operations and Equipment, Depts. of the Army, Air Force, Navy, and Transportation, Washington, DC, 1997.2. M. Burnside, ”BomBots on display at NYC conference,” Times West Virginian, Sep 2006.
3. A. Johnson, A. Klumpp, J. Collier, A. Wolf, ”Lidar-based Hazard Avoidance for Safe Landing on Mars,” AIAA Journal of Guidance, Control and Dynamics, 2002.
4. J. Montgomery, A. Johnson, S. Roumeliotis, L. Matthies. ”The JPL Autonomous Helicopter Testbed: A Platform for Planetary Exploration Technology Research and Development,” Journal of Field Robotics 2006.
5. P. J. Garcia-Pardo, S. S. Sukhatme and J. F. Montgomery, ”Towards Vision-Based Safe Landing for an Autonomous Helicopter,” Robotics and Automated System. 2001.
6. I. Cohen, G. Medioni, ”Detection and Tracking of Objects in Airborne Video Imagery.” CVPR’98Workshop On Interpretation of Visual Motion, pp. 1-8, 1998.
7. S. Xie, Z. Gong, X. Fu, H. Zou, ”Biomimetic Control of Pan- Tilt-Zoom Camera for Visual Tracking Based-on An Autonomous Helicopter,” 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems. pp. 2138-2143, Oct 2007.
8. L. Mejias, S. Saripalli, P. Campoy, G. S. Sukhatme, ”Visual Servoing of an Autonomous Helicopter in Urban Areas Using Feature Tracking,” Journal of Field Robotics, vol. 23, no. 3/4, pp. 185-199, March/April 2006.
9. D. J. Taylor, M. V. Ol, T. Cord, ”SkyTote Advanced Cargo Delivery System,” 2003 AIAA/ICAS International Air and Space Symposium and Exposition: The Next 100 Years, July 2003.
10. S. Saripalli, J. F. Montgomery, and G. S. Sukhatme, ”Visually-guided landing of an unmanned aerial vehicle,” IEEE Transactions on Robotics and Automation, vol. 19, no. 3, pp. 371-381, June 2003.
11. J. Hintze, ”Autonomous Landing of a Rotary Unmanned Aerial Vehicle In A Non-cooperative Environment Using Machine Vision,” Masters dissertation, Dept. Elect. and Comp. Eng., Brigham Young Univ., Provo, UT, 2004.
12. W. Fyfe IV, and R. Johnson, ”Unmanned Tactical Air- Ground Systems Family of Unmanned Systems Experiment,” 2005 IEEE International Workshop on Robots and Human Interactive Communication, pp. 103-108, Aug 2005.
13. P. Oh, ”Integration of Joint-coupling for Visually Servoing a 5-DOF Hybrid Robot,” Ph.D. dissertation, Dept. Mech. Eng., Columbia Univ., New York, NY, 1999.
