admin爬蟲框架簡單學習
一、簡單配置，獲取單個網頁上的內容。
（1）創建scrapy項目
```c
scrapy startproject getblog
```
（2）編輯 items.py
```c
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# /en/latest/topics/items.html
from scrapy.item import Item, Field
class BlogItem(Item):
title = Field()
desc = Field()
```
（3）在spiders 文件夾下，創建 blog_spider.py
需要熟悉下xpath選擇，感覺跟JQuery選擇器差不多，但是不如JQuery選擇器用著舒服（ w3school教程：/mb/reg.asp?kefu=xiaoding ）。
```c
# coding=utf-8
from scrapy.spider import Spider
from getblog.items import BlogItem
from scrapy.selector import Selector
class BlogSpider(Spider):
# 標識名稱
name = blog
# 起始地址
start_urls = [ http://www.cnblogs.com/ ]
def parse(self, response):
sel = Selector(response) # Xptah 選擇器
# 選擇所有含有class屬性，值爲'post_item 的div 標簽內容
# 下麪的 第2個div 的 所有內容
sites = sel.xpath( //div[@ >
items = []
for site in sites:
item = BlogItem()
# 選取h3標簽下，a標簽下，的文字內容 'text()
item[ title ] = site.xpath( h3/a/text() ).extract()
# 同上，p標簽下的 文字內容 'text()
item[ desc ] = site.xpath( p[@ >
items.append(item)
return items
```
（4）運行，
```c
scrapy crawl blog # 即可
```
（5）輸出文件。
在 settings.py 中進行輸出配置。
```c
# 輸出文件位置
FEED_URI = blog.xml
# 輸出文件格式 可以爲 json，xml，csv
FEED_FORMAT = xml
```
輸出位置爲項目根文件夾下。
二、基本的 -- scrapy.spider.Spider
（1）使用交互shell
```c
dizzy@dizzy-pc:~$ scrapy shell http://www.baidu.com/
2014-08-21 04:09:11 0800 [scrapy] INFO: Scrapy 0.24.4 started (bot: scrapybot)
2014-08-21 04:09:11 0800 [scrapy] INFO: Optional features available: ssl, http11, django
2014-08-21 04:09:11 0800 [scrapy] INFO: Overridden settings: { LOGSTATS_INTERVAL : 0}
2014-08-21 04:09:11 0800 [scrapy] INFO: Enabled extensions: TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2014-08-21 04:09:11 0800 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2014-08-21 04:09:11 0800 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2014-08-21 04:09:11 0800 [scrapy] INFO: Enabled item pipelines:
2014-08-21 04:09:11 0800 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6024
2014-08-21 04:09:11 0800 [scrapy] DEBUG: Web service listening on 127.0.0.1:6081
2014-08-21 04:09:11 0800 [default] INFO: Spider opened
2014-08-21 04:09:12 0800 [default] DEBUG: Crawled (200) GET http://www.baidu.com/ (referer: None)
[s] Available Scrapy objects:
[s] crawler scrapy.crawler.Crawler object at 0xa483cec
[s] item {}
[s] request GET http://www.baidu.com/
[s] response 200 http://www.baidu.com/
[s] settings scrapy.settings.Settings object at 0xa0de78c
[s] spider Spider default at 0xa78086c
[s] Useful shortcuts:
[s] shelp() Shell help (print this help)
[s] fetch(req_or_url) Fetch request (or URL) and update local objects
[s] view(response) View response in a browser
# response.body 返廻的所有內容
# response.xpath( //ul/li ) 可以測試所有的xpath內容
More important, if you type response.selector you will access a selector object you can use to
query the response, and convenient shortcuts like response.xpath() and response.css() mapping to
response.selector.xpath() and response.selector.css()
```
也就是可以很方便的，以交互的形式來查看xpath選擇是否正確。之前是用FireFox的F12來選擇的，但是竝不能保証每次都能正確的選擇出內容。
也可使用：
```c
scrapy shell --nolog
# 蓡數 --nolog 沒有日志
```
（2）示例
```c
from scrapy import Spider
from scrapy_test.items import DmozItem
class DmozSpider(Spider):
name = dmoz
allowed_domains = [ dmoz.org ]
start_urls = [ /Computers/Programming/Languages/Python/Books/ ,
/Computers/Programming/Languages/Python/Resources/,
]
def parse(self, response):
for sel in response.xpath( //ul/li ):
item = DmozItem()
item[ title ] = sel.xpath( a/text() ).extract()
item[ link ] = sel.xpath( a/@href ).extract()
item[ desc ] = sel.xpath( text() ).extract()
yield item
```
（3）保存文件
可以使用，保存文件。格式可以 json，xml，csv
```c
scrapy crawl -o a.json -t json
```
（4）使用模板創建spider
```c
scrapy genspider baidu baidu.com
# -*- coding: utf-8 -*-
import scrapy
class BaiduSpider(scrapy.Spider):
name = baidu
allowed_domains = [ baidu.com ]
start_urls = (
http://www.baidu.com/ ,
)
def parse(self, response):
pass
```
本站是提供個人知識琯理的網絡存儲空間，所有內容均由用戶發佈，不代表本站觀點。請注意甄別內容中的聯系方式、誘導購買等信息，謹防詐騙。如發現有害或侵權內容，請點擊一鍵擧報。
0條評論