This document describes how to install Scrapy on Linux, Windows and Mac OS X.
First, you need to install Python, if you haven’t done so already.
Scrapy works with Python 2.5, 2.6 or 2.7, which you can get at http://www.python.org/download/
There are many ways to install Scrapy. Pick the one you feel more comfortable with.
Download Scrapy from the Download page. Scrapy is distributed in two ways: a source code tarball (for Unix and Mac OS X systems) and a Windows installer (for Windows). If you downloaded the tarball, you can install it as any Python package using setup.py:
tar zxf Scrapy-X.X.X.tar.gz
cd Scrapy-X.X.X
python setup.py install
If you downloaded the Windows installer, just run it.
Warning
In Windows, you may need to add the C:\Python25\Scripts (or C:\Python26\Scripts) folder to the system path by adding that directory to the PATH environment variable from the Control Panel.
You can install Scrapy using setuptools‘s easy_install with:
easy_install -U Scrapy
If you’re running Ubuntu 9.10 (or above), use the official Ubuntu Packages, which already solve all dependencies for you and are continuously updated with the latest bug fixes.
If you’re running Debian Linux, run the following command as root:
apt-get install python-twisted python-libxml2 python-pyopenssl python-simplejson
Then:
easy_install -U w3lib
And then follow the instructions in Install Scrapy.
If you are running Arch Linux, run the following command as root:
pacman -S twisted libxml2 pyopenssl python-simplejson
Then:
easy_install -U w3lib
And then follow the instructions in Install Scrapy.
The easiest way to install Scrapy in other Linux distros is through easy_install, which will automatically install Twisted, w3lib and lxml as dependencies. See Installing with easy_install.
Another way would be to install dependencies, if you know the packages in your distros that meets them. See Requirements.
The easiest way to install Scrapy on Mac is through easy_install or pip, which will automatically install Twisted, w3lib and lxml dependencies.
There are two ways to install Scrapy in Windows: