Introduction

Over the years agricultural vehicles become increasingly automated with trajectory row tracking and master-slave vehicle configurations, and autoguided vehicles. Safety is an important aspect. Auto guided vehicles exist in industry, where the surroundings are semistructured and flat. Sopme cars have collision sensors. But in agriculture the ground is not flat.  The vehicles are meant to be driven into crops, and there are certain paths and crops that can be driven in while others cannot. Obstacles include steep hills, lakes, wells and large stones, ditches, wildlife, vehicles and humans. Research has often been done using LIDAR and stereo vision. We suggest a supplement sensor to the fast reacting LIDAR that fuses of high dynamic range stereo and thermography, which will form a feature vector that can classify traversable and non-traversable paths.

Methods

A Claas Axion was mounted two Basler ace 640-90gc colour cameras in a stereo setup with 80cm baseline and a Flir a615 thermal camera. They were hardware synchronized and took three exposures for high dynamic range images. It took only 40ms to acquire each image set. The Flir camera did not have external trigger but with an internal clock at 50Hz (20ms) the software synchronization was accurate to acquire a thermal image within the first two exposures of the stereo cameras. Calibration was done using an A0 sized checkerboard. The thermal camera saw it inverted when doing the acquisition outside where heat from the sky was integrated in the carbon contained in the black print.

Data was acquired for 3 days and a subset containing 2 fields, a yard surrounded with buildings, a public road and gravel roads was selected for analysis. Obstacles included lakes, hedges, stones, vehicles, fences, ramps, concrete barriers, buildings, people hiding in maize, trees, a dog and game birds.

3D reconstruction was done real-time using cuda acceleration and the thermal image was warped to corresponding color pixels. The pixels were partitioned into 600x600mm cells based on the 3D data. A vector of chromaticity, height from a reference plane and temperature was formed for each cell.

Classification of each cell was based on the distribution of these vectors within a cell. The distribution was tested in two ways: 1. Accumulative Histogram Carving. 2. Expectation Maximization fitting with Gaussian Mixture Models. The score was defined as the intersection of a cell histogram and the trained histogram (1), and the Bhattacharyya distance of GMMs (2). It is important to note that using E-M GMM for classifying the distribution of features is different from using normal GMM classification which only classifies features.

Results

The acquisition and synchronization was fast enough that cars driving 80-100 km/h while the tractor was waiting to cross the road showed up on all 3 cameras at matching pixels.

The histogram approach was very, but it required more training than the GMM approach because it does not generalize the data well. Execution speed did not increase with training, because it was carving out distributions that are considered traversable in the same data space. GMM classification was easier to train but was slower in runtime. The more classes it is trained with, the slower it became, but fewer training images were needed.

Results showed that the significant features were Height and Temperature. Obstacles were detected well with chromaticity weights zeroed and equal weight on height and temperature. The temperature was an important feature for detecting living things and vehicles, but also stones and poles. It made it easy to detect people hiding 5 meters inside tall maize, and crouching in tall grass. The height made it possible to see fences, half meter tall concrete barriers versus valid ramps.  The depth resolution was limited at a distance, so small fences and hedges were not detected until within 15 meters.

Lakes and rain pools were tricky to interpret. The water mirror would trick the stereo system into thinking it was farther away. The temperature signal was often colder then surroundings, but it may also reflect the temperature of a building. In cases where a lake was deep compared to the ground around it, the system saw it as an obstacle. Fusion with radar would improve the interpretation of these.

Conclusion

Fusion of stereo and thermography was shown to be a powerful obstacle detector in agricultural settings, where the traversable ground cannot be assumed to be flat. Training can be done by driving the vehicle through traversable paths and crops. However, a vehicle can be pre-initialized with certain knowledge about certain obstacles. Certain classes can also be assigned higher priorities, such as humans.

The system can be used for auto-guided vehicles as well as alert inattentive drivers. The approach can be expanded with other spectral bands (NIR, UV) and waves such as ultrasound, and act complementary to or fused with LIDAR and RADAR systems.