Representation for scene and place recognition

Representation for Scene and Place Recognition

In this project, we propose a representation, PACT (Principal Component Analysis of Census Transoform histograms), for recognizing the semantic category of natural scenes and indoor environments, e.g. forests, coasts, streets, bedrooms, living rooms, etc.

Recognizing places has been attracting many attention in both robotics and computer vision. In some cases we need to recognize the instance of a place (e.g. “room 115?, and in some other cases the semantic category (e.g. “This is an office room.?. Making our robots having the ability to answer all such questions are key to the success of future robotic systems.
We believe that the central problem is a suitable representation ?because places/scenes are so different from objects that we cannot directly borrow representations from object recognition research. We propose PACT, Principal Component Analysis of Census Transoform histograms, as our representation.

Census Transform (CT) summaries local shape and textures by comparing a pixel with its 8 neighbors, such that any 3x3 image region is mapped to one of the 256 discrete cells.

Histogram of CT values is a summarization of local 3x3 shapes. We show that a CT histogram of an image window implicitly encodes global shape, which is essential for recognizing places and scenes.

As shown by this example, we can reconstruct the input image (the left one in each three-image group) requiring that the reconstructed image (the right one in each group). The middle one in each group is the random starting state for the reconstruction algorithm, which try to minimize the difference between CT histogram of the current state and the desired CT histogram.

By further applying PCA (Principal Component Analysis) on the CT histograms, we get the PACT representation.

In summary, PACT has:

superior recognition rate on multiple standard datasets for both place and scene recognition;
Almost no parameters to tune;
Extremely fast evaluation speed (>50 fps).

The source code for PACT is available.

Please note a typo in the README, thanks to Henry Shu for pointing out this error.
Basically, the following lines (one for Swedish leaf and one for scene) are switched.
========================================================================
4. Running the Sweden Leaf example
* 4.2 change main.cpp -- uncomment Load_UIUC_Scenes(), and comment out SwedenLeafContour()
5. Running the scene recognition example
* 5.2 change main.cpp -- comment out Load_UIUC_Scenes(), and uncomment SwedenLeafContour()
======================================================

C++ source code is here. This code is tested under Linux. However, it should also compile and run in Windows.
Please read theREADME.txt in the zip file before you play with the code and executable.

If you use the provided source code, please cite the following paper:
Where am I: Place instance and category recognition using spatial PACT
Jianxin Wu and James M. Rehg
CVPR 2008, Anchorage, Alaska, June 24-26, 2008.