初次接触这个框架,先订个小目标,抓取QQ首页,然后存入记事本。
安装框架(http://scrapy-chs.readthedocs.io/zh_CN/0.24/intro/install.html)
创建一个项目,项目根目录输入
scrapy startproject tutorial
在spiders文件夹下面新建一个qq_spider.py,写入内容如下:
import scrapyclass DmozSpider(scrapy.Spider): name = "dmoz" allowed_domains = ["qq.com"] start_urls = [ "http://www.qq.com/" ] def parse(self, response): filename = response.url.split("/")[-2] with open(filename + ".txt", 'wb') as f: f.write(response.body)
项目根目录输入命令:
scrapy crawl dmoz
抓取到的QQ首页的页面: