Novel 3D Scene Understanding Applications From Recurrence in a Single Image
Shimian Zhang, Skanda Bharadwaj, Keaton Kraiger, Yashasvi Asthana, Hong Zhang, Robert T. Collins and Yanxi Liu, Arxiv Preprint 2022[
PDF][
Bibtex]
Abstract
We demonstrate the utility of recurring pattern discovery from a single image for spatial understanding of a 3D scene in terms of (1) vanishing point detection, (2) hypothesizing 3D translation symmetry and (3) counting the number of RP instances in the image.
Furthermore, we illustrate the feasibility of leveraging RP discovery output to form a more precise, quantitative text description of the scene. Our quantitative evaluations on a new 1K+ Recurring Pattern (RP) benchmark with diverse variations show that visual perception of recurrence from one single view leads to scene understanding outcomes that are as good as or better than existing supervised methods and/or unsupervised methods that use millions of images.
Framework
 |
| Overview of our proposed two-stage method. RESCU is comprised of two primary stages. Stage-I. We extract features from the image to perform unsupervised RP detection and obtain a set of candidate RP(s) and their corresponding region proposals. Stage-II. Candidate RP image crops discovered in Stage-I are passed through a frozen pre-trained feature extractor to obtain their corresponding feature representations. Region proposals derived from the original image are sent through the same extractor to obtain a second set of representations. We apply clustering to the features obtained from the region proposals and RP crops to perform matching and ultimately derive sets of positive and negative samples to train the final RP classifier. The predicted RP(s) are then used in three downstream tasks: RP instance counting, vanishing point detection, and translation detection. We show the discovered information may be used to enhance image captions. |
Translation Symmetry
 |
| Sample results for translation symmetry detection. (a) represents the original images of the translation symmetry ground truth (TS_GT) dataset for which translation symmetry was known to exist. (b) represents rectified outputs of TS_GT images. (c) represents translation symmetry detected in sample images from VPD dataset, and (d) represents rectified outputs of VPD images. |
Caption Enhancement
 |
| Proposed image caption enhancement pipeline. We obtain image captions using OFA. We parse the sentence for collective nouns or a subject noun and use the previously discovered dominant RP count for that corresponding image to add the detected noun's count. We further add discovered VP and TS information to the final caption. |
GRASP Recurring Patterns from a Single View
Jingchen Liu and Yanxi Liu, CVPR2013[PDF][Bibtex]
Abstract
We propose a novel unsupervised method for discovering recurring patterns from a single view. A key contribution of our approach is the formulation and validation of a joint assignment optimization problem where multiple visual words and object instances of a potential recurring pattern are considered simultaneously. The optimization is achieved by a greedy randomized adaptive search procedure (GRASP) with moves specifically designed for fast convergence. We have quantified systematically the performance of our approach under stressed conditions of the input (missing features, geometric distortions). We demonstrate that our proposed algorithm outperforms state of the art methods for recurring pattern discovery on a diverse set of 400+ real world and synthesized test images.
Framework
 |
| An overview of our proposed method: (a) input image; (b) extracted and clustered feature points with top 20 clusters color coded; (c) GRASP optimization framework; (d) automatically discovered recurring pattern after a joint optimization process. |
Optimization
 |
 |
| We design 5 types of moves for iteratively discovering the recurring pattern. |
Greedy Randomized Adaptive Search Procedure (GRASP) for optimization |
Real-World Pattern Distribution
Real-World Image Dataset: [Download]
Acknowledgement
This work is funded partially under NSF grant IIS-1248076 and IIS-1144938
