Segmentation and classification of objects in images is one of the most important and yet one of the most complex problems in computer vision. In this work we propose a new model for natural image object classification using contextual information at the level of image segments. Context modeling is largely independent of appearance-based classification and proposed model enables simple upgrade of existing systems with information from global and/or local context. Context modeling is based on non-parametric use of appearance-based classification results which is a novel approach compared to previous systems that model context on a limited number of rules expressed with a fixed set of parameters. Model implementation resulted in a system that, in our simulations, showed stable improvement of the appearance-based object classification.