‘Part’ly First Among Equals: Semantic Part-Based Benchmarking for State-of-the-Art Object Recognition Systems

Sarvadevabhatla, R K and Venkatraman, S and Venkatesh Babu, R (2017) ‘Part’ly First Among Equals: Semantic Part-Based Benchmarking for State-of-the-Art Object Recognition Systems. Lecture Notes in Computer Science, 10115. pp. 181-197. ISSN 0302-9743

Full text not available from this repository. (Request a copy)


An examination of object recognition challenge leaderboards (ILSVRC, PASCAL-VOC) reveals that the top-performing classifiers typically exhibit small differences amongst themselves in terms of error rate/mAP. To better differentiate the top performers, additional criteria are required. Moreover, the (test) images, on which the performance scores are based, predominantly contain fully visible objects. Therefore, ‘harder’ test images, mimicking the challenging conditions (e.g. occlusion) in which humans routinely recognize objects, need to be utilized for benchmarking. To address the concerns mentioned above, we make two contributions. First, we systematically vary the level of local objectpart content, global detail and spatial context in images from PASCAL VOC 2010 to create a new benchmarking dataset dubbed PPSS-12. Second, we propose an object-part based benchmarking procedure which quantifies classifiers’ robustness to a range of visibility and contextual settings. The benchmarking procedure relies on a semantic similarity measure that naturally addresses potential semantic granularity differences between the category labels in training and test datasets, thus eliminating manual mapping. We use our procedure on the PPSS-12 dataset to benchmark top-performing classifiers trained on the ILSVRC-2012 dataset. Our results show that the proposed benchmarking procedure enables additional differentiation among state-of-the-art object classifiers in terms of their ability to handle missing content and insufficient object detail. Given this capability for additional differentiation, our approach can potentially supplement existing benchmarking procedures used in object recognition challenge leaderboards.

[error in script]
IITH Creators:
IITH CreatorsORCiD
Item Type: Article
Uncontrolled Keywords: Classification (of information); Computer vision; Object recognition; Semantics Error rate; Manual mapping; Object recognition systems; Semantic parts; Semantic similarity measures; Spatial context; State of the art; Test images
Subjects: Computer science > Big Data Analytics
Divisions: Department of Computer Science & Engineering
Depositing User: Team Library
Date Deposited: 17 Apr 2017 10:10
Last Modified: 17 Apr 2017 10:10
URI: http://raiith.iith.ac.in/id/eprint/3171
Publisher URL: http://doi.org/10.1007/978-3-319-54193-8_12
OA policy: http://www.sherpa.ac.uk/romeo/issn/0302-9743/
Related URLs:

Actions (login required)

View Item View Item
Statistics for RAIITH ePrint 3171 Statistics for this ePrint Item