OpenScraperAn open source webapp for scraping: towards a public service for webscraping
Stars: ✭ 80 (+400%)
elves🎊 Design and implement of lightweight crawler framework.
Stars: ✭ 322 (+1912.5%)
BOC FER SpiderUse Scrapy crawl foreign exchange rate from BOC (Bank of China)
Stars: ✭ 18 (+12.5%)
adenineADENINE: A Data ExploratioN PipelINE
Stars: ✭ 15 (-6.25%)
WebSocketPipeSystem.IO.Pipelines API adapter for System.Net.WebSockets
Stars: ✭ 17 (+6.25%)
k8s-knative-gitlab-harborBuild container images with Knative + Gitlab + Harbor inside Kops cluster running on AWS
Stars: ✭ 23 (+43.75%)
GPlayCrawlerNo description or website provided.
Stars: ✭ 47 (+193.75%)
pmanA process management system written in python
Stars: ✭ 14 (-12.5%)
scrapy-distributedA series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.
Stars: ✭ 38 (+137.5%)
dspatchThe Refreshingly Simple Cross-Platform C++ Dataflow / Pipelining / Stream Processing / Reactive Programming Framework
Stars: ✭ 124 (+675%)
restaurant-finder-featureReviewsBuild a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).
Stars: ✭ 21 (+31.25%)
OLX Scraper📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.
Stars: ✭ 15 (-6.25%)
julia-workshop"Integrating Julia in real-world, distributed pipelines" for JuliaCon 2017
Stars: ✭ 39 (+143.75%)
scrapy-cookiesA middleware of cookies persistence for Scrapy
Stars: ✭ 19 (+18.75%)
JustDownlink基于Scrapy+Elasticsearch+Django搭建的分布式电影搜索
Stars: ✭ 28 (+75%)
scrapy facebookerCollection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.
Stars: ✭ 22 (+37.5%)
InstaBotSimple and friendly Bot for Instagram, using Selenium and Scrapy with Python.
Stars: ✭ 32 (+100%)
python-spiderpython爬虫小项目【持续更新】【笔趣阁小说下载、Tweet数据抓取、天气查询、网易云音乐逆向、天天基金网查询、微博数据抓取(生成cookie)、有道翻译逆向、企查查免登陆爬虫、大众点评svg加密破解、B站用户爬虫、拉钩免登录爬虫、自如租房字体加密、知乎问答
Stars: ✭ 45 (+181.25%)
scraping-ebayScraping Ebay's products using Scrapy Web Crawling Framework
Stars: ✭ 79 (+393.75%)
devsearchA web search engine built with Python which uses TF-IDF and PageRank to sort search results.
Stars: ✭ 52 (+225%)
ImageGrabberA Scrapy demo : Download all images from a site
Stars: ✭ 33 (+106.25%)
logparserA tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.
Stars: ✭ 70 (+337.5%)
bgmtoolsBangumi小工具
Stars: ✭ 66 (+312.5%)
codeflareSimplifying the definition and execution, scaling and deployment of pipelines on the cloud.
Stars: ✭ 163 (+918.75%)
IMDB-ScraperScrapy project for scraping data from IMDB with Movie Dataset including 58,623 movies' data.
Stars: ✭ 37 (+131.25%)
allitebooks.comDownload all the ebooks with indexed csv of "allitebooks.com"
Stars: ✭ 24 (+50%)
tibannaTibanna helps you run your genomic pipelines on Amazon cloud (AWS). It is used by the 4DN DCIC (4D Nucleome Data Coordination and Integration Center) to process data. Tibanna supports CWL/WDL (w/ docker), Snakemake (w/ conda) and custom Docker/shell command.
Stars: ✭ 61 (+281.25%)
factoryDocker microservice & Crawler by scrapy
Stars: ✭ 56 (+250%)
scrapy.dartScrapy, a fast high-level web crawling & scraping framework for dart and Flutter
Stars: ✭ 50 (+212.5%)
prime-re.github.ioOpen resource exchange platform for non-human primate neuroimaging
Stars: ✭ 13 (-18.75%)
163Music163music spider by scrapy.
Stars: ✭ 60 (+275%)
scrapy-adminA django admin site for scrapy
Stars: ✭ 44 (+175%)
torchestratorSpin up Tor containers and then proxy HTTP requests via these Tor instances
Stars: ✭ 32 (+100%)
gee🏵 Gee is tool of stdin to each files and stdout. It is similar to the tee command, but there are more functions for convenience. In addition, it was written as go
Stars: ✭ 65 (+306.25%)
animecenterThe source code for animecenter
Stars: ✭ 16 (+0%)
pythonSpider🕷️some python spiders with BeautifulSoup or scarpy
Stars: ✭ 28 (+75%)
invana-botA Web Crawler that scrapes using YAML and python code.
Stars: ✭ 30 (+87.5%)
hk0weatherWeb scraper project to collect the useful Hong Kong weather data from HKO website
Stars: ✭ 49 (+206.25%)
scrapy-zyte-smartproxyZyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
Stars: ✭ 317 (+1881.25%)
proxiProxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.
Stars: ✭ 32 (+100%)