CMU algorithm matches sketches, paintings to photographs
Alexei Efros and his team of cunning robotics researchers at Carnegie Mellon University have developed an image matching algorithm with which computers can identify similar images regardless of medium. Like humans, the system can match sketches and paintings with photographs of similar subjects, and so perform tasks that have traditionally posed problems to machines, such as pairing a simple sketch of a car with a photograph of the same.
Previous methods have relied on images having similar colors, shapes and composition but as assistant research professor of robotics Abhinav Gupta explains, this can result in false positives, with the computer matching, say, outdoor photos with similar proportions of overcast sky, regardless of the specific subject.
The team's algorithm, however, identifies the unique visual elements and properties of an image compared to the others in the pool, then looks for matches on that basis. As a result, their software can match photographs of exterior scenes at different times of year, or match a painting to a photograph of the same subject.
As Efros puts it: "The language of a painting is different than the language of a photograph. Most computer methods latch onto the language, not on what's being said."
The results certainly appear to be impressive. Analyzing a very basic pen sketch of the profile outline of a car, the top matches returned by the algorithm are all profile photographs of cars (all facing the same way as in the sketch). A more detailed sketch of a generic sports car at an oblique angle returns photos of a variety of sports cars taken from about the same angle.
Beyond the obvious applications in image collation and searching, the algorithm may in some instances remove the need for rephotography: the act of literally re-photographing an object after a period of time to compare it to a historical photograph. It could also be used to work out the locations from which paintings of landmarks were made.
The Carnegie Mellon team's research will continue, applying the algorithm to object detection in computer vision, and to speeding up the matching process.
The team's paper, Data-driven Visual Similarity for Cross-domain Image Matching, is available online (along with the source code) and will be presented at SIGGRAPH Asia in Hong Kong on December 14. The research page contains an embedded video in which the results of the algorithm at work can be seen.
Algorithm ya CMU inalingana na michoro, michoro, picha na Alexei Efrenko na timu yake ya watafiti wa roboti katika Chuo Kikuu cha Carnegie Mellon wamebuni algorithm ya kulinganisha picha ambayo kompyuta zinaweza kutambua picha zinazofanana bila kujali vyombo vya habari. Kama binadamu, mfumo unaweza kulinganisha michoro na michoro na picha za mada sawa, na hivyo kufanya kazi ambazo kwa kawaida zimeweka matatizo kwa mashine, kama vile kuunganisha mchoro rahisi wa gari na picha ya gari. "Mafunzo ya awali yamegemea picha zenye rangi, maumbo na muundo sawa, lakini kama profesa msaidizi wa utafiti wa roboti, Abhinav Gupta, anaelezea, hii inaweza kusababisha ""positive za uwongo"" na kompyuta ikilinganisha, kwa mfano, picha za nje na uwiano sawa wa anga lenye mawingu, bila kujali mada maalum." Algorithm ya timu hiyo, hata hivyo, hutambua vipengele vya kipekee vya kuona na mali ya picha ikilinganishwa na wengine katika dimbwi, kisha hutafuta mechi kwa msingi huo. Kwa hivyo, programu yao inaweza kulinganisha picha za mandhari za nje wakati tofauti wa mwaka, au kulinganisha mchoro na picha ya mada moja. "Efras anasema hivi: ""Lugha ya uchoraji ni tofauti na lugha ya picha." Mbinu nyingi za kompyuta hutegemea lugha, si yale yanayosemwa". Kwa hakika matokeo yanaonekana kuwa yenye kuvutia. Kwa kuchambua picha ya msingi ya kalamu ya picha ya gari, mechi za juu zilizorudishwa na algorithm zote ni picha za picha za magari (zote ziko upande mmoja kama katika mchoro). Mchoro wa kina zaidi wa gari la michezo la kawaida katika pembe ya mteremko unarudisha picha za magari mbalimbali ya michezo yaliyochukuliwa kutoka kwa pembe sawa. Zaidi ya matumizi ya wazi katika picha collation na utafutaji, algorithm inaweza katika baadhi ya matukio kuondoa haja ya rephotography: tendo la literally re-photographing kitu baada ya kipindi cha muda kulinganisha na picha ya kihistoria. Pia lingeweza kutumiwa kujua mahali ambapo michoro ya vitu vya kihistoria ilichorwa. Utafiti wa timu ya Carnegie Mellon utaendelea, kwa kutumia algorithm kwa kugundua kitu katika maono ya kompyuta, na kuharakisha mchakato wa kulinganisha. "Kulingana kwa picha ya data-kuendeshwa kwa Data kwa ajili ya ""Mfano wa Kufanana kwa Cross-Domain"" (CDM) ni utafiti wa utafiti wa kisayansi unaofanywa na utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti wa utafiti." Ukurasa wa utafiti una video iliyoingizwa ambayo matokeo ya algorithm kazini yanaweza kuonekana.