HSI-DRIVE

A dataset for the research of hyperspectral image processing applied to autonomous driving systems

Python version of the dataset released (2023/10/06)

Get Started

25-band NIR images recorded under real driving conditions

HSI-Drive is the hyperspectral image (HSI) dataset created by the Digital Electronics Design Group (GDED) of the University of the Basque Country (UPV/EHU). This database is intended to contribute to the research into the use of hyperspectral imaging for the development of advanced driver assistance systems (ADAS) and autonomous driving systems (ADS). The dataset contains a diverse set of images recorded with a small-size 25-band VNIR snapshot camera mounted on a moving automobile. The recordings have been made in different seasons of the year, at different day times, under different weather conditions and on different types of roads. The dataset contains images and videos classified and tagged accordingly to provide rich and diverse data.

...

Why a HSI dataset for ADAS/ADS?

The hypothesis is that the rich spectral information provided by hyperspectral sensors can help develop more robust and more efficient ADS:

  1. More robust: Because spectral information beyond the visible range can be used to better separate objects and backgrounds in challenging driving scenarios due to changing weather and illumination conditions, rapid changes in target appearance, and multiple occlusions among different objects.
  2. More efficient: Because more information at the source should result in systems with fewer processing requirements by simplifying the complex processing pipeline of current image-based scene understanding systems.

...

The dataset

HSI-Drive contains images and video sequences obtained in diverse scenarios and under diverse environmental conditions. All recordings have been made on roads and in towns in the province of Biscay, in the Basque Country, Spain. The dataset is organized according to four parameters:

  1. Season of the year: winter, spring, summer and fall.
  2. Day time: dawn, full daylight and sunset.
  3. Weather: sunny, cloudy, wet/rainy and foggy.
  4. Road type: urban streets and roads, interurban and country roads, and highways.

The first version of the dataset (v1.0) contained 276 annotated images from recordings taken during spring and summer. Version v1.1 fixes some bugs and errors detected in a few ground-truth images of the dataset. Version v2.0 contains 752 images, incorporating new images taken during fall and winter (see the relase notes document).

...

Setup

The recording system setup for this project was extremely simple, consisting of just one Photonfocus MV1-D2048x1088-HS02-96-G2 camera. The Photonfocus MV1 camera is a small-size snapshot camera with a GigEVision interface that can run at up to 42fps depending on its configuration. A 12-bit resolution has been used for raw binary information coding, while the camera throughput has been limited to 11fps to avoid excessive memory consumption. The selected optics was an Edmund Optics 16mm C Series VIS-NIR fixed focal length lens. Attached to the MV1, this lens provides a 30.9º FOV.

...

The sensor

The Imec sensor is a 25-band CMV2K SSM5x5 NIR (600nm-975nm) sensor based on a CMOSIS CMV200 image wafer sensor with 5µmx5µm pixel size and 2048x1088 resolution. The spectral bands are obtained by a mosaic of Fabri-Perot filters that produce 2D images with 5x5 pixel windows. (Images courtesy of IMEC)

example raw video sequence (11.2 fps)

...

Labeling

This dataset is aimed at the development of pixel-level classification systems that directly rely on the separability of the spectral signature of materials and on features obtained from spectral information. The labeling of classes for the image annotation has been performed according to material surface reflectances.

The annotation procedure has been very conservative, manually selecting only the areas that clearly belong to each class, and leaving the edges and some areas of the background unlabeled. This procedure favours that all pixels in a class subset contain only the spectral reflectances of the class concerned. This approach is aimed to maximize ML training based on spectral features to the detriment of techniques that rely on spatial features. It is planned that future versions of the dataset will also incorporate dense semantic annotation files.

Class definitions
      Class1: Road (tarmac)
      Class2: Road marks
      Class3: Vegetation (any kind of vegetation, including wood)
      Class4: Painted metal (road signs, traffic light posts, vehicle bodies etc.)
      Class5: Sky
      Class6: Concrete/stone/brick (sidewalks, walls, facades etc,)
      Class7: Pedestrian/cyclist
      Class8: Water (water courses, puddles etc.)
      Class9: Unpainted metal (back of road signs and signposts, road sign posts, streetlight posts, crash barriers etc.)
      Class10: Glass/transparent plastic (vehicle windscreens, headlights and backlights, windows etc.)

Examples of segmented video recordings

Classes: road/road-marks/vegetation/PEDESTRIANS/sky/others

Classes: road /road-marks /vegetation /PEDESTRIANS /sky /others

Classes: road /road-marks /vegetation /P.METAL /sky /others

Classes: road /road-marks /vegetation /P.METAL /sky /others

Examples of recordings made under challenging ambient conditions

Low lighting and wet weather (6 classes/pedestrians)

Rainy weather with droplets on the camera lens (6 classes/p.metal)

Close to saturated sensor recording in sunny midday (6 classes/pedestrians)

Low frontal sun with strong glares and reflections on partially wet tarmac (6 classes/p.metal)

License terms

HSI-Drive is freely available to academic and non-academic entities for non-commercial purposes as far as they adhere to the following license terms:

  1. The dataset comes “AS IS”, without express or implied warranty. The UPV/EHU does not accept any responsibility for errors or omissions.
  2. A reference to the HSI-Drive dataset has to be included in any work that makes use of the dataset. For research papers, cite our preferred publication(1). For other media cite our preferred publication or the link to the HSI-Drive website.

    3. (1) K. Basterretxea, V. Martínez, J. Echanobe. J. Gutiérrez-Zaballa, and I. del Campo, “HSI-Drive: A Dataset for the Research of Hyperspectral Image Processing Applied to Autonomous Driving Systems”, in proc. of the 32nd IEEE Intelligent Vehicles Symposium, Nagoya, Japan, July 2021.

  3. The distribution of this dataset or modified versions of it is prohibited .
  4. The use of the dataset or of any derivative work for commercial purposes as, for example, licensing or selling the data, or using the data with a purpose to procure a commercial gain is not allowed.

More recent papers

J. Gutiérrez-Zaballa, K. Basterretxea, J. Echanobe, M.V. Martínez, and I. del Campo, "Exploring Fully Convolutional Networks for the Segmentation of Hyperspectral Imaging Applied to Advanced Driver Assistance Systems", in proc. The Workshop of Design and Architectures for Signla and Image Processing (HiPEAC 2022)

J. Gutiérrez-Zaballa, K. Basterretxea, J. Echanobe, M.V. Martínez, U. Martínez-Corral, O. Mata-Carballeira, and I. del Campo, "On-chip hyperspectral image segmentation with fully convolutional networks for scene understanding in autonomous driving", Journal of Systems Architecture, 2023 DOI: 10.1016/j.sysarc.2023.102878

Subscribe to download!

or just send an email to gded@ehu.eus with the subject "download HSI-Drive" and you will receive a password to uncompress the files.

HSI-Drive v1.1

Raw images, HSI cubes, ground-truth images, and false RGBs (3.7 GB)

Videos (SPRING)

20'' example raw video sequences (2.90 GB)

Matlab

Matlab functions that help process the v.1.1 dataset files

HSI-Drive v2.0

Raw images, HSI cubes, ground-truth images, and false RGBs (13.1 GB)

Python version of the dataset

Videos (WINTER)

20'' example raw video sequences (1.6 GB)

Matlab

Matlab functions that help process the v2.0 dataset files