[README in English](https://github.com/knshen/JSearcher/blob/master/README.md)
# JSearcher
JSearcherÊÇÒ»¸ö´¿Java±àÐ´µÄ£¬¿ÉÀ©Õ¹µÄ·Ö²¼Ê½ÅÀ³æ¿ò¼Ü¡£JSearcherµÄÖ÷ÒªÌØÐÔ°üÀ¨£º

* ³£¹æÅÀ³æÓ¦ÓÃµÄ¿ìËÙ¹¹½¨
* ¶à¸öÅÀ³æÈÎÎñ²¢·¢Ö´ÐÐ
* µ¥»ú¶àÏß³ÌÅÀÈ¡
* ¶à¸ö½Úµã·Ö²¼Ê½ÅÀÈ¡
* ¶þ½øÖÆÊý¾ÝµÄÏÂÔØ£ºÈçÍ¼Æ¬ºÍpdfÎÄµµ
* Êý¾Ý´æ´¢£ºÄÚÖÃMysql£¬MongoDB£¬ElasticSearchÈýÖÖ´æ´¢¿ØÖÆÆ÷£¬»òÕßÄãÒ²¿ÉÒÔ¶¨ÖÆ×Ô¼ºµÄ´æ´¢·½Ê½
* ´úÀí·þÎñÆ÷ÉèÖÃ

# °²×°
Ä¿Ç°±¾ÏîÄ¿»¹²»Ö§³Ö´Ómaven°²×°
### Ô´ÂëÏÂÔØ
`git clone https://github.com/knshen/JSearcher`

½øÈëjsearcherÄ¿Â¼£¬ÔËÐÐ`mvn install`£¬ÏÂÔØÏà¹ØÒÀÀµ

### Ìí¼ÓJar°ü
Ìí¼ÓJSearcher.jarµ½Ïà¹ØÏîÄ¿µÄclassPath

# ¿ìËÙ¿ªÊ¼
ÒÔÍøÕ¾[http://quotes.toscrape.com/](http://quotes.toscrape.com/)ÎªÀý£¨ÕâÊÇÒ»¸ö¼ÇÂ¼ÃûÈËÃûÑÔµÄÍøÕ¾£©£¬ÎÒÃÇÏëÒªÅÀÈ¡¸ÃÍøÕ¾ÉÏµÄÃûÑÔ£¬ÅÀÈ¡ÄÚÈÝ°üÀ¨ÃûÑÔÄÚÈÝºÍ×÷Õß¡£¸ÃÀý×ÓµÄÍêÕû´úÂëÎ»ÓÚ[´úÂë](https://github.com/knshen/JSearcher-examples/tree/master/quotes)¡£

### ¶¨ÒåDTO¶ÔÏó
DTO½á¹¹»¯µØ¶¨ÒåÁË´ýÅÀÈ¡µÄÊý¾Ý¡£ÔÚDTOÀà¶¨ÒåÖÐ£¬±ØÐëÎªÊôÐÔ¶¨Òå¶ÔÓ¦µÄsetterºÍgetter£¬·½·¨ÃûÒ²±ØÐëÓëÊôÐÔÃû¶ÔÓ¦¡£
```java
public class QuoteDTO {
    String content;
    String author;
	
    public String getContent() {
        return content;
    }
	
    public void setContent(String content) {
        this.content = content;
    }
	
    public String getAuthor() {
        return author;
    }
	
    public void setAuthor(String author) {
        this.author = author;
    }
}
```

### ¶¨ÒåÊý¾Ý³éÈ¡Æ÷
Êý¾Ý³éÈ¡Æ÷¶¨ÒåÁË´ÓÒ»¸öÒ³Ãæ³éÈ¡Ä¿±êÊý¾ÝµÄ·½·¨¡£Äã¿ÉÒÔÊ¹ÓÃÕýÔÚ±í´ïÊ½£¬JSoup×Ô´øµÄcssÑ¡ÔñÆ÷»òÕßxpathÓï·¨½âÎöÍøÒ³¡£ÏÂÃæµÄ³ÌÐòÊ¹ÓÃÁËJSoupµÄcssÑ¡ÔñÆ÷£º
```java
import java.util.ArrayList;
import java.util.List;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import sjtu.sk.parser.DataExtractor;

public class QuotesExtractor extends DataExtractor {
    @Override
    public List<Object> extract(Document doc, String url) {
        List<Object> data = new ArrayList<Object>();
        List<Element> contents = doc.select("span.text");
        List<Element> authors = doc.select("small.author");
		
        assert contents.size() == authors.size();
		
        for(int i=0; i<contents.size(); i++) {
            QuoteDTO quote = new QuoteDTO();
            quote.setContent(contents.get(i).text());
            quote.setAuthor(authors.get(i).text());
            data.add(quote);
        }
        return data;
    }
}
```
³éÈ¡Æ÷±ØÐë¼Ì³ÐDataExtractorÀà£¬²¢ÇÒÊµÏÖextract·½·¨£¬¸Ã·½·¨·µ»ØÒ»¸öList£¬ListÖÐÃ¿¸öÔªËØÊÇÒ»¸öDTO¶ÔÏó£»²ÎÊýdocÊÇÒ»¸öDocument¶ÔÏó£¬±íÊ¾µ±Ç°ÍøÒ³¶ÔÏó¡£

### ¶¨ÒåÊä³öÆ÷
Êä³öÆ÷¶¨ÒåÁË±£´æÄ¿±êÊý¾ÝµÄ·½·¨¡£Êä³öÆ÷²¢²»ÊÇ±ØÐèµÄ¡£
```java
import java.util.List;
import sjtu.sk.outputer.Outputer;

public class QuotesOutputer extends Outputer {
    @Override
    public boolean output(String task_name, List<Object> data) {
        for(Object obj : data) {
            QuoteDTO quote = (QuoteDTO)obj;
            System.out.println(quote.getContent() + "\n--- " + quote.getAuthor() + "\n");
        }
        return true;    
    }
}
```
Êä³öÆ÷±ØÐë¼Ì³ÐOutputerÀà£¬²¢ÇÒÊµÏÖoutput·½·¨¡£ÔÚoutput·½·¨ÖÐ¿ÉÒÔÒÔÈÎÒâ·½Ê½±£´æÊý¾Ý£¬ÀýÈç±£´æÎªjson£¬xml£¬csv£¬Êý¾Ý¿âµÈ¡£


### ÅÀ³æ²ÎÊýÅäÖÃ
»ù´¡ÅäÖÃÍ¨¹ýyamlÎÄ¼þ¶¨Òå¡£ÅäÖÃÎÄ¼þ¶¨ÒåÁËµ¥¸öÅÀÈ¡ÈÎÎñµÄÒ»Ð©²ÎÊý£¬ÕâÐ©²ÎÊý°üÀ¨£º
* Ïß³ÌÊýÁ¿
* ×î´ó·ÃÎÊÒ³ÃæÊýÁ¿
* ÅÀÈ¡ÈÎÎñÃû³Æ
* DTO¶ÔÏóÂ·¾¶
* ³Ö¾Ã»¯·½Ê½
* ¼¯ÈºÐÅÏ¢
* Êý¾Ý¿â»òÕßESµÄÅäÖÃÐÅÏ¢£¨¿ÉÑ¡£©

¾ßÌåµÄÅäÖÃ·½·¨ºÍÊ¾ÀýÇë¼û[ÅäÖÃÎÄ¼þÊ¾Àý](https://github.com/knshen/JSearcher-examples/blob/master/quotes/quotes.yml)

### Èë¿Ú³ÌÐò
```java
import java.util.ArrayList;
import java.util.List;
import sjtu.sk.scheduler.DefaultScheduler;
import sjtu.sk.scheduler.SpiderConfig;
import sjtu.sk.url.manager.URL;

public class QuotesSpider {
    public static void main(String[] args) {
        //define URL seeds
        List<URL> seeds = new ArrayList<URL>();
        for(int i=1; i<=10; i++)
            seeds.add(new URL("http://quotes.toscrape.com/page/" + i));
		
        //create a single scheduler, load configuration file
        DefaultScheduler ds = DefaultScheduler.createDefaultScheduler("quotes.yml");
        // add extractor and outputer
        SpiderConfig.setDataExtractor(ds, new QuotesExtractor());
        SpiderConfig.setOutputer(ds, new QuotesOutputer());
        // run tasks
        ds.runTask(seeds);
    }
}
```
Ê×ÏÈ±ØÐë¶¨ÒåÖÖ×ÓURLÁÐ±í£¬ÕâÐ©URL½«Ö±½Ó±»Ìí¼Óµ½´ý·ÃÎÊURL¶ÓÁÐÖÐ£»Ëæºó£¬Éú³ÉÅÀ³æµ÷¶ÈÆ÷£¬¼ÓÔØÅäÖÃÎÄ¼þ£¬Ê¹ÓÃSpiderConfigÀàÅäÖÃÊý¾Ý³éÈ¡Æ÷ºÍÊä³öÆ÷£»×îºóÔËÐÐÅÀ³æÈÎÎñ¡£

# Ðí¿ÉÖ¤
[Apache License 2.0](https://github.com/knshen/JSearcher/blob/master/LICENSE)

# ÎÄµµ
´ýÍê³É