Many task domains require robots to interpret and act upon natural language commands which are given by people and which refer to the robot’s physical surroundings. Such interpretation, known as the “grounded language problem,” is challenging both because people, especially untrained users, employ diverse vocabulary and grammar, and because in real settings robots typically have substantial uncertainty about the nature and contents of their surroundings, making it difficult to associate the constitutive language elements (principally noun phrases and spatial relations) of the command text to elements of those surroundings. Symbolic models capture linguistic structure but have not scaled successfully to handle the diverse language produced by untrained users. Existing statistical approaches can better handle diversity, but have not to date modeled complex linguistic structure, limiting achievable accuracy. Recent hybrid approaches have addressed limitations in scaling and complexity, but have not effectively associated linguistic and perceptual features. Our framework, called Generalized Grounding Graphs (G 3 ), addresses these issues by defining a probabilistic graphical model dynamically according to the linguistic parse structure of a natural language command. This approach scales effectively, handles linguistic diversity, and enables the system to associate parts of a command with the specific objects, places, and events in the external world to which they refer. This enables robots to learn word meanings and use those learned meanings to robustly follow natural language commands produced by untrained users. We demonstrate our approach for both mobility commands (e.g. route directions like “Go down the hallway through the door”) and mobile manipulation commands (e.g. physical directives like “Pick up the pallet on the truck”) involving a variety of semi-autonomous robotic platforms, including a wheelchair, a micro-air vehicle, a forklift, and the Willow Garage PR2.
@cite_1: We provide an in-depth description of virtual evidence and its application to Bayesian networks. As we will see, virtual evidence significantly extends the power of Bayesia n networks, and makes them much more widely applicable. It does this by giving Bayesian networks the ability to represent aspects from both directed and undirected graphical models. We describe how both evidence and virtual evidence can be seen as having the evidence nodes in the Bayesian network possess virtual children who are always observed to have value unity, and where the conditional probability table of these children imbue their parents either with a sin gle value (hard evidence) or with a collection of values (soft evidenced), the latter case allowing each such value possessing a weight which gets applied to the probability score. This soft evidence is seen as a straightforward gener alization of evidence, where the soft evidence phenomena is interpreted as information provided from outside the context of the process modeled by the Bayesian network (or alternatively, as an extension to the event space that is par titioned normally by the collection of random variables in the Bayesian network). We discuss Pearl’s example of soft ev idence, interpret hybrid ANN HMM (artificial-neural network hidden Markov model) systems as soft evidence, and conclude that while the form of soft evidence depends only on ratios of scores and not on the scores’ absolute value s, the actual score values may be obtained from any information source so desired. We find that some information sources, however, might be more intuitively justifiable than others.
@cite_0: 
@cite_4: In this work we present an online method for generating topological maps from raw sensor information. We first describe an algorithm to automatically decompose a map into submap segments using a graph partitioning technique known as spectral clustering. We then describe how to train a classifier to recognize graph submaps from laser signatures using the AdaBoost machine learning algorithm. We demonstrate that the we can perform topological mapping by incrementally segmenting the world as the robot moves through its environment, and we can close the loop when the learned classifier recognizes that the robot has returned to a previously visited location.
Other recent work built upon semantic parsing by explicitly considering perception at the same time as parsing . These approaches used limited supervision to learn models that connect language to sets of objects, attributes and relations. Other work used attention-based neural networks to answer questions about images @cite_4 @cite_1 @cite_0 , which do not generally use an intermediate parse. Although uses full supervision to train its semantic parser and grounding classifiers, it is able to additionally learn to understand verb phrases such as pick up'' and put,'' which are challenging because such learning involves parameter estimation and inference over state sequences.