368 Open source projects that are alternatives of or similar to Scrapy Craigslist

A framework for creating semi-automatic web content extractors

Faster requests on Python 3

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Netflix like full-stack application with SPA client and backend implemented in service oriented architecture

Powerful web scraping framework for Crystal

Scape top GitHub repositories and users based on keywords

Go cascadia package command line CSS selector

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉

Web scraper for grabing data from Linkedin profiles or company pages (personal project)

A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, SHFE and news data crawlers on BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Scrapy crawler to collect data on the back catalog of songs listed for sale.

Scrapy Training companion code

Random User-Agent middleware based on fake-useragent

PHP Library for detecting CMS

PHP Scraper - an highly opinionated web-interface for PHP

Fetch user's data across social media

Build a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).

A library that enables programmatic interaction with daft.ie. Daft.ie has nationwide coverage and contains about 80% of the total available properties in Ireland.

MetaData html scraper and parser for Node.js (supports Promises and callback style)

A collection of awesome web scaper, crawler.

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

Scrapy project for scraping data from IMDB with Movie Dataset including 58,623 movies' data.

Scraping Ebay's products using Scrapy Web Crawling Framework

NBA Stats API via Basketball Reference

Scrape, standardize and share public meetings from local government websites

PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

A Router WiFi key recovery/cracking tool with a twist.

Snoop — инструмент разведки на основе открытых данных (OSINT world)

Python + Scrapy + MongoDB . 5 million data per day !!!💥 The world's largest website.

慕课网python分布式爬虫源码-长期更新维护

Node.js framework for modular web scraping and data extraction

HTTP API for Scrapy spiders

A multi-thread crawler framework with many builtin image crawlers provided.

A scrapy spider that scraps times and prices from Voyages Sncf. It uses scrapyrt to provide an API interface.

Nextjs server to query websites with GraphQL

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章

依赖Scrapy和搜狗搜索微信公众号文章

爬取百度贴吧、TapTap、appstore、微博官方博主上的游戏评论(基于redis_scrapy)，过滤器采用了bloomfilter。

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Python入门网络爬虫之精华版

Simple but useful Python web scraping tutorial code.

Scraping publicly-accessible Letterboxd data and creating a movie recommendation model with it that can generate recommendations when provided with a Letterboxd username

基于搜狗微信搜索的微信公众号爬虫接口

python爬虫

Apify actor that crawls Google Search result pages (SERPs) and extracts a list of organic results, ads, related queries and more. It supports selection of custom country, language and location.

A python web scraper for public email lists.

Scrapy middleware to handle javascript pages using selenium

The Python Code Tutorials

A Scrapy Spider for downloading PDF files from a webpage.

A Facebook crawler

Redis-based components for Scrapy.

🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish

Scrapy框架下的pixiv多功能爬虫

Simple Web UI for Scrapy spider management via Scrapyd

[OUTDATED] scrapy spiders to crawl the financial text data 📚 📜 pertinent to train word vectors 🚀

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Seeker - another job board aggregator.

A collection of awesome web crawler,spider in different languages