#### notes

[scrapy hipchat](http://www.hipchat.com/gCvqSX8IC)

```bash
scrapy shell http://www.google.com
```

```python
import inspect
inspect.getmembers(response)
```

run from the project directory (broken_links/broken_links$ )
note that this will append to the items.json file if it exists instead of overriding.

- `rm items.json; scrapy crawl link_spider -o items.json; cat items.json`

- http://doc.scrapy.org/en/latest/topics/settings.html#std:setting-DOWNLOAD_DELAY
- http://doc.scrapy.org/en/latest/topics/link-extractors.html#topics-link-extractors
- http://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.contrib.spiders.Rule

-https://github.com/scrapy/scrapy/blob/e62bbf0766568b902f99d963030e57b96cc2aae6/tests/test_spider.py

```python
Link(url='http://www.github.com/', text='valid github link', fragment='', nofollow=False)
log.msg(repr(links[0]), level=log.INFO)
log.msg('------------------------------- Parsing: %s' % response.url, level=log.INFO)
```

#### notes

[scrapy hipchat](http://www.hipchat.com/gCvqSX8IC)

```bash
scrapy shell http://www.google.com
```

```python
import inspect
inspect.getmembers(response)
```

run from the project directory (broken_links/broken_links$ )
note that this will append to the items.json file if it exists instead of overriding.

- `rm items.json; scrapy crawl link_spider -o items.json; cat items.json`

- http://doc.scrapy.org/en/latest/topics/settings.html#std:setting-DOWNLOAD_DELAY
- http://doc.scrapy.org/en/latest/topics/link-extractors.html#topics-link-extractors


> "rules
Which is a list of one (or more) Rule objects. Each Rule defines a certain behaviour for crawling the site. Rules objects are described below. If multiple rules match the same link, the first one will be used, according to the order they’re defined in this attribute."
>
> - http://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.contrib.spiders.Rule
 


-https://github.com/scrapy/scrapy/blob/e62bbf0766568b902f99d963030e57b96cc2aae6/tests/test_spider.py

```python
Link(url='http://www.github.com/', text='valid github link', fragment='', nofollow=False)
log.msg(repr(links[0]), level=log.INFO)
log.msg('------------------------------- Parsing: %s' % response.url, level=log.INFO)
```

Add quote from docs about rule order
Add quote from docs about rule order