http://chestnutheng.github.io/tags/scrapy/ Recent content in Scrapy on 子恒的博客 Hugo zh-CN Wed, 20 Mar 2019 23:00:30 +0800 http://chestnutheng.github.io/scrapy/ Wed, 20 Mar 2019 23:00:25 +0800 http://chestnutheng.github.io/scrapy/ <p>scrapy是个很简单强大的python爬虫框架，不需要处理网络相关逻辑就可以轻松爬取。本文介绍了一些基本的内容。需要注意的是，本文档中所有的●表示非必须内容。<br> 本教程基于官方文档：<a href="https://scrapy-chs.readthedocs.io/zh_CN/0.24/topics/settings.html">官方文档</a></p> <h1 id="安装scrapy">安装scrapy</h1> <h2 id="centos">CentOS</h2> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span><span class="lnt">2 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">yum install python34-devel.x86_64 </span></span><span class="line"><span class="cl">pip3 install scrapy </span></span></code></pre></td></tr></table> </div> </div><h1 id="创建项目">创建项目</h1> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span><span class="lnt">2 </span><span class="lnt">3 </span><span class="lnt">4 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">//创建项目hello </span></span><span class="line"><span class="cl">scrapy startproject hello </span></span><span class="line"><span class="cl">//创建一个爬虫（在项目根目录运行，不要加http://），名字为baidu，域名为www.baidu.com </span></span><span class="line"><span class="cl">scrapy genspider baidu &#34;www.baidu.com&#34; </span></span></code></pre></td></tr></table> </div> </div><p>● 需要注意的是，如果要无视robots.txt文件，请在下面的<code>settings.py</code>中设置<code>ROBOTSTXT_OBEY = False</code><br> ● 刚才建好的项目目录文件树如下：</p> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt"> 1 </span><span class="lnt"> 2 </span><span class="lnt"> 3 </span><span class="lnt"> 4 </span><span class="lnt"> 5 </span><span class="lnt"> 6 </span><span class="lnt"> 7 </span><span class="lnt"> 8 </span><span class="lnt"> 9 </span><span class="lnt">10 </span><span class="lnt">11 </span><span class="lnt">12 </span><span class="lnt">13 </span><span class="lnt">14 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">tree hello </span></span><span class="line"><span class="cl">hello </span></span><span class="line"><span class="cl">|-- hello </span></span><span class="line"><span class="cl">| |-- __init__.py </span></span><span class="line"><span class="cl">| |-- items.py // 输出结构体定义 </span></span><span class="line"><span class="cl">| |-- middlewares.py </span></span><span class="line"><span class="cl">| |-- pipelines.py </span></span><span class="line"><span class="cl">| |-- __pycache__ </span></span><span class="line"><span class="cl">| |-- settings.py // 设置，记得把 ROBOTSTXT_OBEY = False </span></span><span class="line"><span class="cl">| |-- spiders </span></span><span class="line"><span class="cl">| |-- __init__.py </span></span><span class="line"><span class="cl">| |-- baidu.py // 刚创建的爬虫 </span></span><span class="line"><span class="cl">| |-- __pycache__ </span></span><span class="line"><span class="cl">|-- scrapy.cfg </span></span></code></pre></td></tr></table> </div> </div><h1 id="shell测试代码">Shell：测试代码</h1> <p>使用shell可以帮助实现爬虫的代码，查看网页的相关信息。结合浏览器中的审查元素查看会让你构造出需要选择的数据部分。</p>