Test Dataset Download Instructions

Download the test dataset from here: IDDAW Test Images

Download Instructions

Click to download the Dataset and more details. This will redirect you to the registration page to download the dataset.

Register an account at IDD Insaan website.

Go to Dataset > Download page in the menu.

Download the IDD-AW Dataset by clicking on 'Download' under 'IDD AW Dataset'. This will generate a 24 hour token to download the dataset.

Extract the downloaded compressed file into a folder.

IDD-AW Dataset page
Download here

The dataset, including training data, ground truth, and an evaluation script, will be made publicly available post-competition in accordance with ICPR guidelines.

Evaluation

Heirarchical Labeling and Tree Distance

The essence of SmIoU lies in the introduction of hierarchical penalty, a strategy that takes into account the semantic relationships between classes. Heirarchical Labeling refers to the structured organization of classes into a tree-like hierarchy, capturing semantic relationships and dependencies between them in the context of semantic segmentation. It provides a inherent semantic relationship between classes and also provides structured heirarchy for a comprehensive understanding of classes.

Misclassifications within critical classes, and non-critical classes classified as critical, are penalized based on their distance in the class hierarchy. The tree distance (td) between a pair of labels is the length of the shortest path in the class hierarchy tree divided by 2

Calculation of Safe mIoU

The calculation of SmIoU involves computing the individual safe ious for each class. The safe IoUs are the IoUs with a penalty for misclassification of that class weighted by the tree distance. The final SmIoU score is obtained by taking the mean of these individual safe ious.

Let C be the set of all classes at the bottom level of the hierarchy, d(c,s) be the tree distance between class c and s, and n be the number of levels in the hierarchy, gt_c the set of pixels in the ground truth with label c and pred_s the set of pixels in the prediction with label s . We define the following quantities:

Now we define SmIoU as follows:

$$\label{eq:1} I_c^{\text{safe}} = \begin{cases} \begin{aligned}[t] I_{c,c} & - \sum_{s \in C , s \neq c}\frac{d(c,s)}{n}I_{c,s}^{\text{safe}} & \text{ if } c \in C_{\text{imp}} \\ I_{c,c} & - \sum_{s \in C_{\text{imp}} }\frac{d(c,s)}{n}I_{c,s}^{\text{safe}}& \text{ else. } \end{aligned} \end{cases}$$

$$\label{eq:2} \text{SmIoU} = \frac{\sum_{c\in C} I_c^{\text{safe}}}{|C|}$$

Code for calculating SmIoU

Download the IDD-AW Dataset

First create separate directories for groundtruth and your predictions

Clone the code from IDDAW kit into your local dir

Install the requirements using pip install -r requirements.txt

Just run the command python safe_iou.py --gt_dir gt_dir --pred_dir your_preds_dir