GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → scrapinghub → scmongo

scrapinghub / scmongo

Licence: other
MongoDB extensions for Scrapy

Programming Languages

python
139335 projects - #7 most used programming language

scmongo

This project contains some extensions for using MongoDB with the Scrapy web crawling framework.

Requirements

  • Scrapy 0.14 or above
  • pymongo 2.4 or above

Install

Download and run: python setup.py install

Available extensions

Mongo Cache Storage

Module: scmongo.httpcache

A MongoDB backend for HTTP cache storage. It stores responses using GridFS.

To use it, set the following Scrapy setting in your project:

HTTPCACHE_STORAGE = 'scmongo.httpcache.MongoCacheStorage'
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].