Top 615 crawler open source projects

(deprecated) 🐱 koshort is a Python package for Korean internet spoken language crawling and processing... or maybe Korean domestic cat.

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

A Soccer Dashboard created by scraping EPL website using Akka backend and ReactJS frontend and IBM Cloudant for object storage. IBM Cloud Foundry is used to host both frontend and backend app.

fetchman is a simple crawler system/简单好用的爬虫框架

군바리 친구들을 위한 자동으로 편지 보내기

Multiplexing web client supporting HTTP/2 and WHATWG URL compliant parser written in C

超高速异步协程Python爬虫

📷 Tistory photo grabber

SEO & Security Audit for Websites. Lighthouse & Security Headers crawler, Sitemap/Keywords/Images Extractor, Summarizer, etc ...

WebHunger is an extensible, full-scale crawler framework that supports distributed crawling, aiming at getting users focused on web page parsing without concerning for the crawling process.

方便的从浏览器复制浏览器头

Scraping the real estate agencies for up-to-date house listings as soon as they arrive!

Riichi Mahjong Kit: (1) Game log crawler (sqlite3, json, bs4); (2) Game log preprocessor; (3) Deterministic algorithms library

A web service that turns an arbitrary web page into structural JSON data and easy-to-use APIs with just a few clicks

高校高校统一身份认证 Node.js 优雅可扩展示例，已集成今日校园签到(支持多平台一键部署)